ENH: set multi-index names as NamedTuples

[dr-leo]

import pandas as PD
s1=PD.Series((1,4,9,16), name=dict(a=1,b=2))
s2=s1*2
s2.name=dict(c=5,d=7)
s1
0 1
1 4
2 9
3 16
Name: {'a': 1, 'b': 2}, dtype: int32
df = PD.concat((s1,s2), axis=1) # this should raise ValueError as the resulting df is broken
df.shape # ok, this works...
(4, 2)
df # But this doesn't.
<repr(<pandas.core.frame.DataFrame at 0x35af770>) failed: TypeError: unhashable type: 'dict'>
df.columns # This is not a proper index...
Index([{'a': 1, 'b': 2}, {'c': 5, 'd': 7}], dtype='object')

Comments:

1. There may be other ways aside from concat that lead to broken df's if a Series.name attribute is unhashable.
2. If Series.name is a dict, one could try to convert it into a namedtuple. That's hashable.

[jorisvandenbossche]

Shouldn't we just disallow that a Series name is unhashable?

[dr-leo]

Good proposal. This would make the API clear and consistent. It should
also be less error-prone than catching all situations in which a
Series.name is propagated to an index.

And one can always use namedtuples instead of dict to store
multi-dimensional key-value pairs as metadata in Series.name or
df.columns. This could be mentioned somewhere in the docs, I think, as
namedtuple is not so widely used, although it comes in handy.

The following is a bit off-topic, but I am kind of struggling with
building a nnametuple class factory returning a singleton, i.e. don't
create a new namedtuple class if the same has been created before. I
found a verbose and hardly elegant solution in pandaSDMX... So I was
wondering if pandas could provide a utility function for this common task...

Am 13.09.2014 21:11, schrieb Joris Van den Bossche:

Shouldn't we just disallow that a Series name is unhashable?

—
Reply to this email directly or view it on GitHub
#8263 (comment).

[jreback]

@dr-leo pull-request for this?

I think you can simply intercept the __setattr__ call (and validate) in core/generic.py

[dr-leo]

Do we agree what we want?

Option 1: require Series.name to be hashable as suggested by Joris
2. enforce that df.columns is hashable.

As I wrote, I favour option 1. But your hints for a PR seem to imply
option2.

Am 14.09.2014 00:49, schrieb jreback:

@dr-leo https://github.com/dr-leo pull-request for this?

I think you can simply intercept the |setattr| call (and validate)
in core/generic.py

—
Reply to this email directly or view it on GitHub
#8263 (comment).

[dr-leo]

@jeff:

I am unfamiliar with the PD sources. But I've seen that Series inherits
from core.generic.PDFrame. So your hint makes sense even for option 1,
i.e. preventing Series.name to be set to an unhashable type.

I will look into this and if I succeed, send my first PR. But due to
other projects it will take a couple of weeks. I see no urgency. This is
likely for v0.16.

@ALL: for most of you this is a matter of a couple of minutes. So if you
feel this should be done earlier, feel free to jump in.

Leo

Am 14.09.2014 00:49, schrieb jreback:

@dr-leo https://github.com/dr-leo pull-request for this?

I think you can simply intercept the |setattr| call (and validate)
in core/generic.py

—
Reply to this email directly or view it on GitHub
#8263 (comment).

[dr-leo]

I have just submitted a minimalist PR addressing the easier part of this issue, i.e., validating that Series.name is hashable.

[dr-leo]

Some thoughts:

Consider the following script:

from numpy.random import randn
import pandas as PD
import pdb
idx = PD.MultiIndex.from_product((('one', 'two'), ('foo', 'bar')), names = ('level1', 'level2'))
df=PD.DataFrame(randn(4,4), columns = idx)
pdb.set_trace()
s=df.one.bar

1. The purpose of this issue is this: When extracting a column from a multi-indexed df, the returned series should not just have the immediate key at the lowest index level, but a namedtuple whose field names are the MI level names and whose items are the keys at each MI level.
2. Tracing through the execution of a command such as s=df.one.foo shows what one would expect: The Series at df.one.bar is retrieved by walking down the MI levels. Here we have two levels. df.foo yields a 2-column df with one level and keys foo and bar. The first index level (one, two) is forgotten as it is not part of the sub-df. The second iteration yields the actual Series with name 'bar'. 3. If we want to set the Series.name to the namedtuple as described above, we need to remember the level names and the labels. How? Where?
3. I find it inconsistent to attach to the Series the full path through the levels (i.e. Level1:one, Level2:bar, whereas df.one only yields a df with a simple index ('foo', 'bar'). df.one should have a multi-index as well having just 'one' at Level1 and ('foo','bar') at Level2. This would reflect the idea of attaching the corresponding subset of the multi-index to whatever subset of columns we extract from a df. In other words: Extracting a sub-frame or Series yields a multi-index with the same number of levels as the original frame, but with a smaller subset of tuples. For Series, the single tuple is stored in a namedtuple whereas a sub-frame will have a proper MultiIndex.
   Does this semantics make sense? This would break some code, sure. But is the Series.name as namedtuple without also changing the behavior for sub-frames convincing?

[toddrjen]

I think it should use similar mechanics to now, except with the addition of
a name. So it should return the same number of values as it does now. In
cases where it currently drops a value, it should continue to do so. In
cases where it doesn't currently drop a value, the value should still be
maintained.

[dr-leo]

Just to make sure I understand what you mean:

Do you agree that all NDFrames (Series, DataFrame, Index, MultiIndex and
perhaps Panel) should have a name attribute?

I think this would be a good idea as it would make things more
consistent and provide a place for simple metadata at DataFrame or Index
level. So you could name a MultiIndex "Earth" and the levels continents,
countries, regions etc. A DataFrame with this MultiIndex could be named
'GDP'. And df.America.Canada would have a 1-level column index with
labels for the regions. Its name would be a namedtuple with two fields
from the original DF (Continent, Country). If you pick the series for
the region Quebec, the Series.name would have three fields: Continent,
Country, Region.

This would not break any code.

Am 17.01.2015 um 09:16 schrieb toddrjen:

I think it should use similar mechanics to now, except with the
addition of
a name. So it should return the same number of values as it does now. In
cases where it currently drops a value, it should continue to do so. In
cases where it doesn't currently drop a value, the value should still be
maintained.

—
Reply to this email directly or view it on GitHub
#8263 (comment).

[toddrjen]

So names from successive slices should be appended to the existing name?
What if the existing name is a string or ordinary tuple?