How to note that some methods aren't available for lazy dataframes?

[MarcoGorelli]

Here are some examples which don't work raise with polars lazyframes:

• DataFrame.shape (the length isn't, and can't be, known)
• Column.__len__ (same as above)
• comparisons (e.g. df1 == df2, but there may be a way around this)
• anything column-related (as there's no polars.LazyColumn - but there may be a way around this)

I'd like to think we can work round points 3 and 4. But not 1 and 2, not really sure how that could work

I know we want everything to be independent of the execution engine, but how do we plan for these to work for lazy engines?

[betatim]

There was a discussion in the Array API repo a while back that was somewhat similar. It was about things like __bool__ and __float__ and how to deal with these for lazy array libraries. data-apis/array-api#642 (comment) concluded that it was Ok that if a user accessed those a computation was forced.

From the point of view of someone using the dataframe API to write a tool that consumes dataframes (lazy or eager) you don't want to have to distinguish in your code whether a dataframe is lazy or not. It makes me think that if a user wants to know the shape or len, then they want to know the shape/len now. So a lazy dataframe would have to go away and evaluate enough to be able to answer that query. A bit like with bool(some_array), I want to know the answer now, not at some point later.

[rgommers]

@MarcoGorelli what does "don't work" mean? Does it force evaluation, raise an Exception, or ....? I think either choice is valid - e.g., Dask will force evaluation when needed, while the comment @betatim linked to had an implementation that needed to be 100% lazy and hence had to raise.

Also, I think DataFrame.shape() is not a good example - I don't see a reason why that cannot remain lazy. The reason bool(), int() et al. couldn't stay lazy is because the Python language itself will not allow returning anything other than actual bool/int/etc. instances. For .shape(), it's perfectly okay to return a lazy object.

[rgommers]

xref gh-195 for other lazy-implementation-specific questions

[MarcoGorelli]

Apologies, by "doesn't work" I meant that it raises an exception

I think I'd be -1 on forcing computation for .shape (or anything else). Right now, if you have a polars.LazyFrame, then everything is guaranteed to be practically zero-cost up until you hit .collect(). If some operations can trigger potentially very expensive operations, then that guarantee is broken

So the alternatives would be:

• raise for .shape
• "For .shape(), it's perfectly okay to return a lazy object."

For the latter, what would the lazy object be? A lazy tuple? Such a thing doesn't exist in polars. If it's something which exists only in the standard, then presumably there needs to be some way to materialise it?

Anyway, I think this highlights that there's need for discussion on this topic. But that's what we're here for! Fortunately it's in the agenda for tonight's call

I think my general stance is:

• no method should force computation for lazy engines
• some methods may raise if used on lazy objects, and we should document this

[rgommers]

For the latter, what would the lazy object be? A lazy tuple? Such a thing doesn't exist in polars.

It's not hard to build though, so I don't think there's a fundamental problem there. Polars can add it if desired.

If it's something which exists only in the standard, then presumably there needs to be some way to materialise it?

The standard does not have anything that's specified as being lazy. It only has a tuple return type, with the understanding that it may be any other object that duck types as a tuple.

Materialization is library-specific. I'd expect that if a Polars dataframe has a .collect(), then any other lazy objects that may exist in Polars would also have a .collect() (but if it's something else, then that's equally fine).

[MarcoGorelli]

Sure - and if polars didn't want to add LazyFrame.shape, we'd be OK with just raising NotImplementedError in its standard-compliant implementation?

[kkraus14]

Alternatively for things like shape and length which are data dependent, we could do something like allow returning NULL or -1 or something to indicate that the value is unknown?

For object equality comparisons I'm not sure they're even a part of the standard yet? But in general for APIs that return Scalars that could possibly be used for control flow what would be the experience for a lazy implementations if only an explicit API can force materialization? I.e. what would the experience of doing something like if df.get_column_by_name('a').any(): ...?

[MarcoGorelli]

For object equality comparisons I'm not sure they're even a part of the standard yet?

there is DataFrame.__eq__:

dataframe-api/spec/API_specification/dataframe_api/dataframe_object.py

Lines 283 to 300 in b03bb4f

	def __eq__(self, other: DataFrame | Scalar) -> DataFrame: # type: ignore[override]
	"""
	Compare for equality.
	Nulls should follow Kleene Logic.
	Parameters
	----------
	other : DataFrame or Scalar
	If DataFrame, must have same length and matching columns.
	"Scalar" here is defined implicitly by what scalar types are allowed
	for the operation by the underling dtypes.
	Returns
	-------
	DataFrame
	"""
	...

[MarcoGorelli]

But in general for APIs that return Scalars that could possibly be used for control flow what would be the experience for a lazy implementations if only an explicit API can force materialization? I.e. what would the experience of doing something like if df.get_column_by_name('a').any(): ...?

Yeah it's tricky. I really think this needs discussion and design

Staying with polars as an example - indeed, what would df.get_column_by_name('a').any() return?

• In the eager case, it returns a Python scalar
• in the lazy case, we could say that it returns some lazy scalar

But crucially, there doesn't exist a lazy scalar in polars, and I don't think there would be much appetite for adding one.

We could work around this by returning a LazyScalar. And we could give it a .collect method, for compatibility with the rest of polars. But LazyScalar isn't part of the polars docs, so users wouldn't know what to do with it. And if no .collect method is part of the Standard either, then concretely what could users do with such a LazyScalar?

This is why I'm suggesting that some methods raise NotImplementedError for lazy implementations, and that we should document for which ones that would be appropriate.
For example, we could say that DataFrame.sum must always be supported by any standard-compliant implementation, but that DataFrame.shape is allowed to raise for lazy engines

[MarcoGorelli]

Comment for reference:

Lazyframe don't have a formalized shape. I don't think they should support [equality comparison]

pola-rs/polars#10274 (comment)

[mcrumiller]

For the latter, what would the lazy object be? A lazy tuple?

This opens up a whole can of worms if we want to have lazy versions of future-materializable items. Lazy scalars? Does df.lazy().select('a')[0] return a materializable?

[MarcoGorelli]

In polars that would raise (as it would in the dataframe standard - which, just for reference, is the repo this issue is in, just in case you came here by accident 😄 And if not, welcome, good to have you!)

Today's call touched on a lot of topics, including forced materialisation in operations used in control flow, user-experience, methods like collect

There wasn't a concrete conclusion (other than that this needs more discussion), but we did say we want to punt on it and get it working for eager libraries

I'd suggest that for now we at least document what might methods might have implementation-dependent behaviour for lazy engines, so that if you want to write library-agnostic code, then as long as:

• you know your inputs are going to be eager (say, because that's what you document that you support);
• OR you only use the methods which are not marked as implementation-dependent for lazy engines.

then you know your code will run fine

[MarcoGorelli]

Something I did want to share, with regards to forced materialisation, was this post by Liam Brannigan: https://www.linkedin.com/feed/update/urn:li:activity:7079836009755996160?utm_source=share&utm_medium=member_desktop

One way to do this in Polars and Pandas is the .upsample method. This method inserts the missing time steps.

However, upsample isn't part of the lazy API in Polars and so we have to exit lazy mode to do upsample in eager mode.

But...we can do this another way where we stay in lazy mode and can still stream large datasets.

We do this by first defining a DataFrame with a time column that has all the time steps. We then do a left join with the gappy dataframe to get the data. With the rows added we can then handle the missing values - in this case by interpolating the values column.

The good news is that by re-thinking how we would do this in Pandas we can stay lazy and use streaming for large datasets.

By not implementing something in lazy mode, polars forced him to rethink his approach, and to write more efficient code.

If, conversely, polars had materialised the data under-the-hood for him, then he would have ended up write less-efficient-than-possible code

[betatim]

Also, I think DataFrame.shape() is not a good example - I don't see a reason why that cannot remain lazy. The reason bool(), int() et al. couldn't stay lazy is because the Python language itself will not allow returning anything other than actual bool/int/etc. instances. For .shape(), it's perfectly okay to return a lazy object.

I agree that Python "forced our hand" for bool and co. However, I'd still have preferred the current outcome even if Python hadn't forced us. The reason being that I think being able to write array/dataframe consuming code without having to differentiate lazy/eager (in control flow) is a big bonus. If you can get it.

Maybe shape() isn't a good example, maybe I"m not creative enough to think of use-cases for it (or similar things) where you would access it and not immediately want to use the actual value. You can and should discuss whether or not a user should be accessing it, because it causes computation, but users be users. Sometimes they want to do things even if they are costly.