groupby sorting - don't specify, or forbid?

[MarcoGorelli]

From the last call, the cuDF folks stated that a goal of theirs is to let users prototype with pandas on CPU, and then switch to GPU with cuDF and have their code "just work".

One issue they currently face is groupby: pandas sorts by default, so that's what their users expect

Even if pandas were to adopt the standard in its main namespace, this would not accomplish their goal. This is because whilst the standard doesn't specify whether groupby should sort, it doesn't forbid it either - thus, pandas could continue sorting in groupby by default whilst respecting the standard, but then cuDF would be no better off.

cuDF also said that if pandas were to implement the standard in a separate namespace, then users wouldn't necessarily want to look up the standard and would expect things to just work as they're used to (i.e. pandas main namespace).

The only way I can think of of accomplishing cuDF's goal, without changes in cuDF, would be:

1. the Standard forbids sorting in groupby
2. pandas adopts the standard in its main namespace

I don't mean to be a "Pessimistic Pete", but the second one seems unlikely to land. Some deviations from pandas in cuDF may be warranted.

My suggestion is:

• for cuDF's end-user groupby issue, they could remove the default from groupby, thus forcing users to specify a value for sort. Users would be required to type an extra 2 words, but they'd probably be better off because of it
• the Standard stays developer-oriented, rather end-user-oriented
• to minimise surprise to users of the Standard, that the standard explicitly forbid sorting in groupby. It's been mentioned that developers might only test their code using pandas and then expect it to work with other DataFrame libraries, so this would reduce the chances of surprises. And so long as it's in a separate namespace, there's no risk of breaking millions of users' code

EDIT: on the last point - maybe the standard doesn't need to forbid sorting, but the pandas implementation of it shouldn't sort by default

[MarcoGorelli]

cc @shwina @kkraus14 for visibility (I don't know if you receive notifications from this repo, so tagging you just in case)

[shwina]

Thanks, @MarcoGorelli -- I agree with everything you've said here.

I do think that a user-facing dataframe API has much more value than a purely developer-facing API. And while I recognize that Pandas will likely always provide a superset of that API, I also think that it should be at minimum an implementation of the standard, and as flexible an implementation of the standard as possible.

So I think this would be the best outcome here:

maybe the standard doesn't need to forbid sorting, but the pandas implementation of it shouldn't sort by default

[MarcoGorelli]

Hey - when I wrote

the pandas implementation of it shouldn't sort by default

I meant if pandas were to implement the standard in the separate namespace - it would be too disruptive of a change for the pandas main namespace.

I think this is probably the opposite of what you want - in which case, would making sort a required argument in cudf.groupby be feasible?

[kkraus14]

• to minimise surprise to users of the Standard, that the standard explicitly forbid sorting in groupby. It's been mentioned that developers might only test their code using pandas and then expect it to work with other DataFrame libraries, so this would reduce the chances of surprises. And so long as it's in a separate namespace, there's no risk of breaking millions of users' code

EDIT: on the last point - maybe the standard doesn't need to forbid sorting, but the pandas implementation of it shouldn't sort by default

It would be great if the standard explicitly made no promises or guarantees regarding the output ordering of operations like groupby and didn't include a sort parameter in the groupby api. Implementers of the standard could choose to sort or provide ordering semantics in any way that they wish within the details of their implementation, but the standard shouldn't require or guarantee any of that behavior.

[MarcoGorelli]

Sure, but if the standard makes no promises about groupby sorting, then there's nothing stopping pandas from keeping its current groupby behaviour.

And then that doesn't solve your issue that people come to cuDF expecting to also sort in groupby, which you said you've like to change

[jbrockmendel]

My takeaway from last week is that "The Standard" is completely irrelevant and all anyone really cares about is the pandas behavior, where cudf wants an API change. The place to ask for that is on the pandas tracker, and the way to convince people is to argue it will benefit pandas users.

[kkraus14]

And then that doesn't solve your issue that people come to cuDF expecting to also sort in groupby, which you said you've like to change

From my perspective, the challenge is that there's a parameter with a default value and documented behavior from Pandas as opposed to it being an implementation detail. I.E. the docs for the sort parameter say:

Sort group keys. Get better performance by turning this off. Note this does not influence the order of observations within each group. Groupby preserves the order of rows within each group.

This dictates a few different things:

• Default behavior is to sort by the grouping keys with sort=True
• Pandas guarantees that the input dataframe order is maintained within each group for "payload" values regardless of the sort value
• The fact that the input dataframe order is maintained for grouping keys when sort=False isn't documented behavior
  • It's unclear to me if this is an implementation detail or if this is guaranteed behavior from Pandas

In an ideal world, the API we're trying to define wouldn't have a sort keyword and would explicitly make no promises regarding the ordering of grouping keys or "payload" values as many dataframe implementations don't / can't support these behaviors.

[kkraus14]

My takeaway from last week is that "The Standard" is completely irrelevant and all anyone really cares about is the pandas behavior, where cudf wants an API change. The place to ask for that is on the pandas tracker, and the way to convince people is to argue it will benefit pandas users.

The conversation from last week was a bit all over the place, but my main takeaway was that the API we're trying to build now is targeted at library / applications developers much moreso than end users and that the first dataframe implementation of the standard that library / applications developers will build against will likely be Pandas. This leads to a concern that anywhere that Pandas supports more than the standard, that developers may use those features while thinking they are targeting the standard. They may then be disappointed when they don't get the implementation portability they were looking for.

For targeting end users, we threw around a lot of frustrations, ideas, concerns, and more. A lot of that surrounded the existing inertia of Pandas and how we can give Pandas users a happy path towards alternative dataframe implementations that are mainly offering enhanced performance and scalability as compared to Pandas. I agree that if our conversations lead down to "we want to change Pandas" then those conversations should then move to the Pandas issue tracker. I don't think we're there yet and have just really started the "end user" conversations.

[MarcoGorelli]

In an ideal world, the API we're trying to define wouldn't have a sort keyword and would explicitly make no promises regarding the ordering of grouping keys or "payload" values as many dataframe implementations don't / can't support these behaviors.

OK, sure. Let's say the standard doesn't have a sort keyword.

Pandas would then implement a superset of the API (like Ashwin said) and would keep its sort keyword with its current behaviour.

How does this help cuDF?

[kkraus14]

OK, sure. Let's say the standard doesn't have a sort keyword.

Pandas would then implement a superset of the API (like Ashwin said) and would keep its sort keyword with its current behaviour.

How does this help cuDF?

If it's all going to be in the main namespace then I think ultimately it causes problems and makes the work we're doing here less valuable. I think it would be an unreasonable expectation to have people double checking all of their work to see if they're using parameters that are in Pandas but aren't in the standard. People would likely just continue writing Pandas code as they always have and other dataframe implementations will continue to get pressure to match Pandas API and behaviors.

If it's a separate namespace and Pandas doesn't implement the sort keyword in that namespace then I think that there would be enough justification to get library / application developers interested in having dataframe portability to not be reliant on the implementation details of Pandas with regards to its ordering behaviors.

To be clear, this is beyond just cuDF. My personal experience is with cuDF, Dask (both as a producer and consumer of dataframe APIs), Spark, Arrow, and Ibis (again both as a producer and consumer), all of which can be multi-threaded and/or distributed. This often makes following existing behaviors of Pandas challenging and a significant detriment to performance in many situations.

[MarcoGorelli]

Sorry, I'm confused, and I feel like I'm hearing contradictory statements.

On the call, Ashwin said

You're right in that maintaining two namespaces is not tenable for the long run. The real canary is pandas. What pandas does, everyone will generally follow. Ideally, pandas would exclusively support the standard API. My hope is that, even if not 1,2,3 years, that eventually it would happen.

And here, you're saying that you want a separate namespace in which groupby doesn't have a sort keyword.

Which is it? Do you want the main pandas namespace to change so that the transition to cuDF is smoother, or do you want a separate namespace in which everything can be compatible from the get-go?

(I'm not sure how my tone is coming across, so just to be clear: I'm perfectly calm, and don't mean anything I've written in an aggressive/sarcastic way. I'm genuinely confused 😄 , and trying my best to make sense of what we're trying to achieve here)

[kkraus14]

On the call, Ashwin said

You're right in that maintaining two namespaces is not tenable for the long run. The real canary is pandas. What pandas does, everyone will generally follow. Ideally, pandas would exclusively support the standard API. My hope is that, even if not 1,2,3 years, that eventually it would happen.

And here, you're saying that you want a separate namespace in which groupby doesn't have a sort keyword.

Which is it? Do you want the main pandas namespace to change so that the transition to cuDF is smoother, or do you want a separate namespace in which everything can be compatible from the get-go?

I can't speak for Ashwin.

If there isn't a separate namespace in Pandas, I could only imagine having some package or something to validate you're following the standard and not using functions / keywords that aren't part of the standard. Without that, it will put undue burden on library / application developers to build against the standard effectively. I completely understand and think it's completely reasonable if the Pandas community did not want to support a separate namespace.

In an ideal world purely from the cuDF and other libraries perspectives, Pandas only supports the standard API and all dataframe implementations only support the standard API so users get full portability. In the real world, that isn't in Pandas users' best interests and isn't a feasible thing to do.

(I'm not sure how my tone is coming across, so just to be clear: I'm perfectly calm, and don't mean anything I've written in an aggressive/sarcastic way. I'm genuinely confused 😄 , and trying my best to make sense of what we're trying to achieve here)

Your tone has been perfectly cordial 😄. I appreciate the pushback to make sure that we're actually going to drive impact as opposed to being a perfect example of the xkcd comic (https://xkcd.com/927/).

[jbrockmendel]

I applaud @MarcoGorelli for not being a pessimistic Pete but I have no such compunction. We are getting nowhere and designing an API no one will use. This is a waste of time and we should cut our losses. Put a bow on what we have now and declare this endeavor a rousing success.

[aregm]

@kkraus14 @jbrockmendel @MarcoGorelli
I want to reiterate the original goals when we started this WG:

• Provide developer-facing APIs to standardize among many dataframe library developers
• Keep it minimal and simple
• Agree on the core parts and do not rathole on the disagreements - document them and provide reasonable expectations and exclusions.

So, IMHO, not the perfect standard is better than no standard. We should not pick on something small, like sort, as there are more significant issues to agree upon.

For Pandas - I think having a standard package -> Pandas namespace -> main namespace path is good enough, as long the transition from the first step to the second is not years. From my team side, I commit to helping build, support, and maintain the Standard API namespace in Pandas.

For other libraries, cuDF, Modin, Vaex, Koalas, etc., targeting performance sort is less critical, but trying to be compatible with Pandas can be a problem, so Pandas as a big bear can keep the current semantics, as there are tons of code already written, meanwhile stating that sort is not compliant, but you can use pandas.standard_df.sort for compliant one.
We can't produce an ideal standard that will satisfy everyone - C++ is a good example. It took implementers ~5 years to accept ARM, ~10 years to produce the first standard from ARM, and another ~10 to bring everyone to the same turf. And there are still incompatibilities. Luckily, we are wiser and less fragmented, so I suggest focusing on pushing the standard out.

[jbrockmendel]

FYI @aregm is my boss at Intel. By complete coincidence, I 100% agree with him. [An imperfect] standard is better than no standard. So let's punt on sorting and be done.