Iterating over a column

[MarcoGorelli]

In the last meeting, there was a section when we talked some default which, if pandas changed, would make things easier for other libraries.

One example which @kkraus14 gave, at about 0:43:30, was about iterating over a column in Python-land, value by value.

@shwina mentioned that polars doesn't allow this.

I may have misunderstood what Keith was referring to, but to me it looks like it is allowed?

In [77]: df = pl.DataFrame({'ts': pl.date_range(datetime(2020, 1, 1), datetime(2020, 1, 10))})

In [78]: for i in range(len(df)):
    ...:     print(f'value at position {i} is: {df["ts"][i]}')
    ...:
value at position 0 is: 2020-01-01 00:00:00
value at position 1 is: 2020-01-02 00:00:00
value at position 2 is: 2020-01-03 00:00:00
value at position 3 is: 2020-01-04 00:00:00
value at position 4 is: 2020-01-05 00:00:00
value at position 5 is: 2020-01-06 00:00:00
value at position 6 is: 2020-01-07 00:00:00
value at position 7 is: 2020-01-08 00:00:00
value at position 8 is: 2020-01-09 00:00:00
value at position 9 is: 2020-01-10 00:00:00

Could you please clarify what it is that polars doesn't allow, which you would like it if pandas disallowed?

EDIT: the above example requires

import polars as pl
from datetime import datetime

to run

[kkraus14]

@MarcoGorelli you're using a __getitem__ call with a scalar as opposed to an __iter__. That being said it does look like polars allowing iterating over a column. cuDF on the other hand explicitly poisons the method because there's no way to do it reasonably on the GPU.

import polars as pl
import cudf
from datetime import datetime

pl_df = pl.DataFrame({'ts': pl.date_range(datetime(2020, 1, 1), datetime(2020, 1, 10))})
cudf_df = cudf.DataFrame.from_arrow(pl_df.to_arrow())  # cudf supports everything above, I'm just lazy

# This works for polars
for val in pl_df['ts']:
    print(val)

# This loudly errors that Series object is not iterable.
for val in cudf_df['ts']:
    print(val)

[MarcoGorelli]

There's also

In [104]: df = pl.DataFrame({'ts': pl.date_range(datetime(2020, 1, 1), datetime(2020, 1, 10))})

In [105]: iter_ = df.iter_rows()

In [106]: next(iter_)
Out[106]: (datetime.datetime(2020, 1, 1, 0, 0),)

In [107]: next(iter_)
Out[107]: (datetime.datetime(2020, 1, 2, 0, 0),)

In [108]: next(iter_)
Out[108]: (datetime.datetime(2020, 1, 3, 0, 0),)

[kkraus14]

There's also

In [104]: df = pl.DataFrame({'ts': pl.date_range(datetime(2020, 1, 1), datetime(2020, 1, 10))})

In [105]: iter_ = df.iter_rows()

In [106]: next(iter_)
Out[106]: (datetime.datetime(2020, 1, 1, 0, 0),)

In [107]: next(iter_)
Out[107]: (datetime.datetime(2020, 1, 2, 0, 0),)

In [108]: next(iter_)
Out[108]: (datetime.datetime(2020, 1, 3, 0, 0),)

cuDF poisons the iterrows() method as well

[rgommers]

We discussed this today, and concluded that we should implement __iter__ in the standard and make it raise an exception, to avoid a construct that's easy to write but very slow. Just leaving it out isn't good enough, because Python will fall back to __getitem__ and make a for-loop work.

The rationale for raising is that there's almost always much better ways of achieving whatever the user needs to achieve, rather than for-loops. And if really needed, there's still the explicit __getitem__ usage within a for-loop.

[MarcoGorelli]

Sorry, just coming back to this

because Python will fall back to getitem and make a for-loop work.

If the standard doesn't have getitem, then would poisoning __iter__ actually be necessary?