BUG? Indexing + assignment does not work if member selected after indexing

[dengemann]

Hi there,

this is one that has puzzled me for quite some time, but somehow it never asked about it. It seems whether an conditional assignment to a df column is performed or not depends on the order of indexing / member getting inside the left-hand expression of the assignment statement. See this snippet:

import pandas as pd

# create some data
data = {'foo': list('abcde'),
        'bar': range(5)}
df = pd.DataFrame(data)

# create arbitrary index.
my_idx = df.bar > 2
print df[my_idx]

# assign to a  member which was accessed after indexing.
df[my_idx].foo = 'X'

# see there is no change
print df

# assign to member which was accessed before indexing.
df.foo[my_idx] = 'X'

# now it has worked
print df

Is there a deliberate reason for this behaviour? My feeling about this is that it may confuse new users and may be hard to track down, as this goes silently and looks like correct code.

[jreback]

You are assigning to a copy of the object, use .loc or .ix

In [40]: df = pd.DataFrame(data)

In [41]: df.loc[df.bar>2,'bar'] = 5

In [42]: df
Out[42]: 
   bar foo
0    0   a
1    1   b
2    2   c
3    5   d

Since you are chaining operators, you are responsible for deciding whether to mutate the object or not (and whether to get a reference to the object)

e.g. you are free to do

df_new = df[df.bar>2]
df_new.bar = 5

[dengemann]

Yes, makes sense, I was assuming a quite implicit reference / value issue.
But if it's just about chaining, why does the order make a difference here? In my examples apparently the access member early variant creates a reference, the access member late variant creates a copy.
My gut feeling is that both versions should consistently create either a reference or a copy. Or is this a common language design pattern I did not know about (would be quite implicit ...)? For example in R the order does not matter and both versions would pass.

If so, maybe we could add a note somewhere in the docs pointing out that the 'pandathonic' way of conditional assignment is using .ix / .loc and that the order matters.

[jreback]

df.bar returns a Series that as its data a view onto the dataframe (its easy because that is how they are stored, and not a copy. This is for performance reasons (easy to see as you would often index off this and do stuff, no reason to have a copy).

df[df.bar>2] can be a view (into the underlying numpy data), but is not always. This is generally related to how the data is stored.

Pandas philosophy is a to have user perform explicit mutating operations, and not accidently set other things.

Chaining by definition means its much harder to figure out what is going on.

Feel free to do a PR to explain this case in more detail (might go in regular docs + 10min.rst).

Python itself allows things like this, but numpy has the same basic issue, for performant reasons you will often get a view; setting that can sometimes trigger a copy, so effects are somewhat non-deterministic.

I think the view (pun!) was that allow common operations (e.g. [] to slice on the rows of a frame) to succeed even though [] is the column access operator (equiv to df.bar and returns a view on the object).

[dengemann]

Python itself allows things like this, but numpy has the same basic issue, for performant reasons you will often get a view; setting that can sometimes trigger a copy, so effects are somewhat non-deterministic.

Thanks for detailing this, I already noticed this in a strict numpy context but always accepted it as some sort of peculiarity. The reason I came up with it here is, presumably, because I most often stumbled across this in the pandas context where I often write such kinds of statements.

I think the view (pun!) was that allow common operations (e.g. [] to slice on the rows of a frame) to succeed even though [] is the column access operator (equiv to df.bar and returns a view on the object).

Yes makes sense, I always wondered about the internals of the DataFrame [] operator ... ;-)
I hope the indexing API will evolve a bit slower / stabilise in the future. I remember having had some trouble writing cross-version-compatible pandas code. And it's kind of a weird moment where you find yourself indexing the underlying numpy structures instead of using the pandas API ...

Feel free to do a PR to explain this case in more detail (might go in regular docs + 10min.rst).

I think this is a good idea ;-)

[jreback]

in 0.11, after lots of clamor, we introduced the .loc for location label based and .iloc for positional based indexing, to hopefully reduce confusion. See http://pandas.pydata.org/pandas-docs/dev/10min.html#selection, and http://pandas.pydata.org/pandas-docs/dev/indexing.html#basics. If you have any comments, a PR is welcome, or let me know :)

Also if you think a feature is needed, pls open an issue

[dengemann]

in 0.11, after lots of clamor, we introduced the .loc for location label based and .iloc for positional based indexing, to hopefully reduce confusion.

I noticed that and I do appreciate the choice taken ;-)
FYI I was referring to tiny a pandas API I contributed here:

http://martinos.org/mne/auto_examples/export/plot_epochs_as_data_frame.html#example-export-plot-epochs-as-data-frame-py

to make neurophysiology accessible data the pandas way. Setting this up together with a cross version-API-compliant example took some time (related to cross-version testing).

If you have any comments, a PR is welcome, or let me know :)

Sure I'll let you know. I'm about to refactor / add to the pandas interface in the next days, I'm sure I'll find something worth PRing / discussing ;-)

[jreback]

going to put this for 0.12 and can think about it...thanks

[jreback]

I belive this is now well-documented as chaining, and mostly resolved AFAIK by keeping weak-reference to the cacher ..
@dengemann pls open a new issue if you find anything weird!