DOC: df.iterrows() returns view for homogenous columns, and copy for heterogenous / perf warning

[shashispace]

perf discussion: http://stackoverflow.com/questions/24870953/does-iterrows-have-performance-issues/24871316#24871316

import pandas as pd
df = pd.DataFrame({'a':range(5),'b':range(5)})
print df

for _, row in df.iterrows():
    row.a += 1
print df

df['c'] = 'what'

for _, row in df.iterrows():
    row.b += 1
print df

   a  b
0  0  0
1  1  1
2  2  2
3  3  3
4  4  4

[5 rows x 2 columns]
   a  b
0  1  0
1  2  1
2  3  2
3  4  3
4  5  4

[5 rows x 2 columns]
   a  b     c
0  1  0  what
1  2  1  what
2  3  2  what
3  4  3  what
4  5  4  what

[5 rows x 3 columns]

[jreback]

and? lots of pandas operations do this
more efficient this way

[shashispace]

Hmmn... that's probably true. But it leads to newbies wasting lots of time... anyways, I thought consistency should be preferred over speed.

[jreback]

this is a very common idiom on python in general; iterating over a list-like. You should NEVER modify something you are iterating over. this is NOT a copy in python space either (though I think it IS allowed to be a copy, maybe in python 3 it is, not sure).

So maybe a doc warning in that section would be good, informing users that setting values using ANY of the iteration methods IS NOT a good idea.

can you do a pull-request?

[shashispace]

So why not make it consistent and return a copy even for homogenous dataframes? (I guess you've already said that it is more efficient.)

I had this issue with something I was running - and I couldn't figure out what was going on, cause a toy example I constructed behaved differently (cuase all its columns were same type) from the main code. I think others would have this issue too - it would be best to make it consistent between the two cases. I have nothing against returning a copy.

Sorry, I don't know enough programming to create a proper pull request... embarrassing, I know. I should dive into this stuff. It would be good for me... any tips on how to get started?

[cpcloud]

Check the wiki there are a lot of tips on contributing

[jreback]

@shashispace well performance is worth it not to make a copy when people do iterrows (which FYI, is never necessary as there are much better methods). so a doc warning would be in order but that's about it.

[shashispace]

Sounds good. Thanks for indulging me. What are some of the better ways to iterate? I was doing:
for i in df.index:
df.ix[i,'a'] += 1

(that's a bad example, but you get the idea - I need to pick the column inside the loop...) I found this way considerably slower, though I didn't benchmark it or anything.

[jreback]

you don't iterate or use loops, use a mask instead, e.g.

df.ix[df['a']>5,'a'] += 1

the df['a']>5 produces a mask, see here: http://pandas-docs.github.io/pandas-docs-travis/indexing.html#boolean-indexing

[shashispace]

I think I didn't explain it well, let me try again:

I have dataframe with large number of columns, and I need to add a number to a different column in each row. In row one, I might add to 'a', in row 13, I might need to add to column 'b' etc.

Writing this, I realize that I can iterate over the columns.. which would be way more efficient and I can write vectorized code for that. I can't remember if matlab allowed writing to random columns in different rows, is there a vectorized way to do that in pandas?

Thanks so so much - greatly appreciate your replies. It's definitely motivated me to get more involved than just using pandas.

[jreback]

pandas aligns, so just create the rhs with the values and it will work, e.g. something like

s = Series([1,2,3],index=[10,12,20])
df.loc[:,['a','c']] += s

would add the indicates values from s in columns a and c

so easiest to do this column by column, of course depending how you are determining what you are adding in the first place.

might be even easier to simply construct a frame that you are adding, e.g.

frame_to_add = DataFrame(a = Series([1,2,3],index=[10,20,30]), c = Series([1,2,3],index=[4,5,6]))
df += frame_to_add.fillna(0)

does it all at once

[shashispace]

awesome. thanks again.