PERF: Should 'iter[x]'s be generators?

[max-sixty]

Looking here: https://github.com/pydata/pandas/pull/11325/files#diff-1e79abbbdd150d4771b91ea60a4e1cc7R686, #11325

...currently itertuples is building a new full objects in memory and then creating an iterator over those objects. Because it's building a new tuple for each row, these aren't views (I think). Should these instead yield a tuple / row / item on each call, using a generator?

The upside is that you don't need to rebuild the object in memory at once. The downside is that it is dependent on the underlying data (as though it were a view).

(and possible there is something going on that I don't understand, either with pandas or generators...)

[jreback]

how so? These are created one by one and they yield each result. These are cross sections and not possible to be views at all (aside from the fact that they are tuples and not ndarrays)

[jreback]

@MaximilianR you are referring to a pull-request btw that has not been reviewed

[max-sixty]

@jreback:
The code is currently creating a list of all the tuples, and then iterating over them. This could be done with a generator so it doesn't need to create the whole list upfront. Here's a quick outline. It doesn't yet include the index.

def itertuples(self):
    for i in range(len(self.index)):
        yield tuple(self.iloc[i, :])

The current PR is orthogonal, apologies for the confusion. Here's the line in master: https://github.com/pydata/pandas/blob/master/pandas/core/frame.py#L676

And a quick test of the code:

In [30]: df
Out[30]: 
          0         1         2
0  0.915169  0.948095  0.046362
1  0.571498  0.705196  0.574511
2  0.247967  0.087300  0.050543
3  0.766040  0.084670  0.722872
4  0.506722  0.730061  0.884221
5  0.933631  0.990122  0.224536
6  0.950318  0.111794  0.781591
7  0.079632  0.110314  0.727207
8  0.210386  0.300626  0.840557
9  0.513826  0.178803  0.519423

In [31]: df.itertuples()
Out[31]: <zip at 0x1077daa08>

In [36]: list(df.itertuples(index=False))
Out[36]: 
[(0.91516922497785835, 0.94809513572151194, 0.046362440951684736),
 (0.57149809527904927, 0.70519573519318379, 0.57451121996442733),
 (0.24796728352013642, 0.087299660770303267, 0.050542964228438292),
 (0.76603960250622294, 0.084669515793419414, 0.72287226941419858),
 (0.50672198977616179, 0.73006078739715474, 0.88422141062737925),
 (0.93363109313640236, 0.99012212400605759, 0.22453609421051479),
 (0.95031758155074642, 0.11179412831821223, 0.7815908189291001),
 (0.079631728382511091, 0.11031385076955258, 0.72720691312650887),
 (0.21038626136450145, 0.30062582061343091, 0.84055670019872053),
 (0.51382555167081712, 0.17880347816615116, 0.51942313327047096)]



In [50]: def itertuples(self):
        for i in range(len(self.index)):  
                yield tuple(self.iloc[i, :])
   ....:         

In [52]: itertuples(df)
Out[52]: <generator object itertuples at 0x10759ccf0>

In [53]: itertuples(df).__next__()
Out[53]: (0.91516922497785835, 0.94809513572151194, 0.046362440951684736)


In [51]: list(itertuples(df))
Out[51]: 
[(0.91516922497785835, 0.94809513572151194, 0.046362440951684736),
 (0.57149809527904927, 0.70519573519318379, 0.57451121996442733),
 (0.24796728352013642, 0.087299660770303267, 0.050542964228438292),
 (0.76603960250622294, 0.084669515793419414, 0.72287226941419858),
 (0.50672198977616179, 0.73006078739715474, 0.88422141062737925),
 (0.93363109313640236, 0.99012212400605759, 0.22453609421051479),
 (0.95031758155074642, 0.11179412831821223, 0.7815908189291001),
 (0.079631728382511091, 0.11031385076955258, 0.72720691312650887),
 (0.21038626136450145, 0.30062582061343091, 0.84055670019872053),
 (0.51382555167081712, 0.17880347816615116, 0.51942313327047096)]

[max-sixty]

OK, both iteritems and iterrows are generators, and that's where the copying might happen. itertuples uses numpy's machinery to its advantage, and is much faster than my suggestion...

Closed, from my POV.