BUG: Panel.to_frame() with MultiIndex major axis #5417

TomAugspurger · 2013-11-02T19:37:15Z

Closes #5402 WIP for now.

So this is a hack to get things "working".

I mainly just wanted to ask if there was a better way to get the levels and labels to the MultiIndex constructor.
Don't spend long thinking about it (this is my PR after all), but if you know of a quick way off the top of your head, I'd be interested.
I essentially need to

flatten the MultiIndex from major axis (levels and labels)
combine that flattened index with the minor axis (levels and labels)

Step 2 could be complicated by minor_axis being a MultiIndex also, I should be able to refactor this pretty easily to handle that. I think my was is making several copies of each index, and I'm not sure that they're all necessary.

jtratner · 2013-11-02T21:28:36Z

What do you mean by flatten? You can get tuples by using the values
property. Check out _sparsify too.
On Nov 2, 2013 3:37 PM, "Tom Augspurger" notifications@github.com wrote:

Closes #5402 #5402 WIP for now.

So this is a hack to get things "working".

I mainly just wanted to ask if there was a better way to get the levels
and labels to the MultiIndex constructor.
Don't spend long thinking about it (this is my PR after all), but if you
know of a quick way off the top of your head, I'd be interested.
I essentially need to

flatten the MultiIndex from major axis (levels and labels)

combine that flattened index with the minor axis (levels and labels)

Step 2 could be complicated by minor_axis being a MultiIndex also, I
should be able to refactor this pretty easily to handle that. I think my
was is making several copies of each index, and I'm not sure that they're

all necessary.

You can merge this Pull Request by running

git pull https://github.com/TomAugspurger/pandas to-frame-multi

Or view, comment on, or merge it at:

#5417
Commit Summary

BUG: Panel.to_frame() with MultiIndex major axis

File Changes

M pandas/core/panel.pyhttps://github.com/BUG: Panel.to_frame() with MultiIndex major axis #5417/files#diff-0(36)

M pandas/tests/test_panel.pyhttps://github.com/BUG: Panel.to_frame() with MultiIndex major axis #5417/files#diff-1(19)

Patch Links:

https://github.com/pydata/pandas/pull/5417.patch

https://github.com/pydata/pandas/pull/5417.diff

TomAugspurger · 2013-11-04T00:53:23Z

OK, I think this is just about there if you want to take a look. Essentially I check if the major and minor axes are a MultiIndex. If they are, I repeat labels the necessary number of times and rearrange them appropriately. I need to write a few more test cases to make sure I'm not getting any false positives.

I didn't see a vbench covering to_frame. Think we need one?

jtratner · 2013-11-04T00:58:40Z

I believe we need to set up a way to have vbench conditional on version -
@y-p I recall you saying that vbenches were supposed to run across many
versions of pandas, but maybe we could add a version check (decorator?
kwarg?) to the setup code?

jtratner · 2013-11-04T01:00:21Z

pandas/tests/test_panel.py

@@ -1320,6 +1320,51 @@ def test_to_frame_mixed(self):
        # Previously, this was mutating the underlying index and changing its name
        assert_frame_equal(wp['bool'], panel['bool'], check_names=False)

+    def test_to_frame_multi_major(self):


standard notes on testing MI: make sure you have 1 example that is non-lex-sorted and 1 that has nan in some of the levels.

jreback · 2013-11-04T01:02:00Z

maybe put those conversions in a function (in index?)

jtratner · 2013-11-04T01:04:19Z

pandas/core/panel.py

-                           names=[maj_name, min_name], verify_integrity=False)
+        if isinstance(self.major_axis, MultiIndex):
+            major_levels = self.major_axis.levels
+            major_labels = map(lambda x: np.repeat(x, K), self.major_axis.labels)


why not make this a list comprehension instead?

ghost · 2013-11-04T01:38:43Z

@jtratner, that was re: test_perf.py, I had some scripts relying on it to auto. sample perf back in history
and find regression points using bisection. When some contrib. changes to test_perf broke
it for older pandas, I added some comments to it to make it clear that back-compat is a requirement,
since test_perf itself relies on pandas for data processing.

I didn't see anything here that calls for changes in vbench, can you explain the issue?

jtratner · 2013-11-04T01:40:47Z

okay, that makes sense. I just wasn't sure if to_frame() has always been a method on Panel. thanks for clarifying.

TomAugspurger · 2013-11-04T01:50:48Z

@jreback Which conversions? Just this line?

major_labels = [np.repeat(x, K) for x in self.major_axis.labels]

ghost · 2013-11-04T01:52:19Z

@jtratner, ok well In that case I think the existing start_date kwarg attached to each vbench should be all you need.

jtratner · 2013-11-04T01:53:08Z

wow, I should have noticed that.

jreback · 2013-11-04T02:00:30Z

pandas/core/panel.py

+            major_names = [major_names]
+        if isinstance(self.minor_axis, MultiIndex):
+            minor_levels = self.minor_axis.levels
+            minor_labels = map(lambda x: np.repeat(x, N), self.minor_axis.labels)


@TomAugspurger I meant these if/else....they are basically the same for the minor/major axes so could do it with a function; even better to have this as a index method, something like (which does essentially what your if else does), except it works separately on Multi-index (the if part) and Index (the else part)

def conform_to_shape(self, N, K): return levels, labels, names

jreback · 2013-11-04T02:34:58Z

makes a lot of sense as an index method because has method in index and multi index

TomAugspurger · 2013-11-09T02:24:56Z

pandas/core/index.py

@@ -2373,6 +2373,42 @@ def format(self, space=2, sparsify=None, adjoin=True, names=False,
        else:
            return result_levels

+    def to_hierarchical(self, N, K=1):


Should I rename this to _to_hierarchical?

TomAugspurger · 2013-11-09T17:52:21Z

Ok those tests should be clearer now. Earlier @jtratner mentioned

I'm not totally sure, but I believe you could actually pay the cost of filtering the panel on selector first, and then you wouldn't have to filter on selector everywhere else. Slower, but cleaner maybe?

which makes a lot of sense. I just need to figure out the best way filter the panel... It's not immediately obvious to me.

TomAugspurger · 2013-11-09T19:34:31Z

Can you post some examples of 'selectors' - - might be easier to reason
about the selector separate from the context of this method.

selector is a 1-d array of bools built by raveling a 2-d mask. The mask is the (major, minor) axis pairs that contain NaNs (or will contain NaNs).

Example:

idx = pd.MultiIndex.from_tuples([(1, 'one'), (1, 'two'), (2, 'one'), (2, 'two')])
df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': ['a', 'b', 'c', 'd']}, index=idx)
wp = pd.Panel({'i1': df, 'i2': df})

wp.loc['i1', (1, 'one')]['A'] = np.nan
In [24]: mask = com.notnull(wp.values).all(axis=0)

In [25]: mask
Out[25]: 
array([[False,  True],
       [ True,  True],
       [ True,  True],
       [ True,  True]], dtype=bool)
In [28]: selector = mask.ravel()

In [29]: selector
Out[29]: array([False,  True,  True,  True,  True,  True,  True,  True], dtype=bool)

so that one false is coming from the nan. It means we want to filter the first (major, minor) pair from the output.

TomAugspurger · 2013-12-25T16:49:38Z

EDIT: I think this is a bug. made an issue at #5773

Getting back to this, and I'm cleaning up some setting with copy errors in my tests.

Should the following work?

        idx = MultiIndex.from_tuples([(1, 'one'), (1, 'two'), (2, 'one'),
                                      (2, 'two')])
        df = DataFrame([[1, 'a', 1], [2, 'b', 1], [3, 'c', 1], [4, 'd', 1]],
                       columns=['A', 'B', 'C'], index=idx)
        wp = Panel({'i1': df, 'i2': df})

        wp.loc['i1', (1, 'one'), 'A'] = np.nan

Raises an

IndexError                                Traceback (most recent call last)
<ipython-input-2-3346cc0dd38b> in <module>()
      5 wp = Panel({'i1': df, 'i2': df})
      6 
----> 7 wp.loc['i1', (1, 'one'), 'A'] = np.nan

/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.13.0rc1_87_g9fefe8c-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.pyc in __setitem__(self, key, value)
     94             indexer = self._convert_to_indexer(key, is_setter=True)
     95 
---> 96         self._setitem_with_indexer(indexer, value)
     97 
     98     def _has_valid_type(self, k, axis):

/Users/tom/Envs/pandas-dev/lib/python2.7/site-packages/pandas-0.13.0rc1_87_g9fefe8c-py2.7-macosx-10.8-x86_64.egg/pandas/core/indexing.pyc in _setitem_with_indexer(self, indexer, value)
    297                 obj = self.obj[item]
    298                 index = obj.index
--> 299                 idx = indexer[:info_axis][0]
    300                 try:
    301                     if idx in index:

IndexError: tuple index out of range

TomAugspurger · 2014-01-03T15:03:50Z

@jreback ready to go at your leisure. let me know if you want a note in v0.14.0.txt. I put one in release.rst.

jreback · 2014-01-03T15:27:11Z

pandas/core/panel.py

+        else:
+            minor_levels = [self.minor_axis]
+            # Anyone think of a better way to do this? np.repeat does not
+            # do what I want


anyway you can do a simple function like construct_multi_parts for the non-mi case? so don't have repeated code?

Yeah, not sure why I didn't before...

jreback · 2014-01-03T18:22:17Z

@TomAugspurger this is a bug fix, so unless you want to, nothing needed for whatsnew (fyi now targeting 0.13.1, though it still may go to 0.14, depends if we have critical bugs ..)

jreback · 2014-01-03T19:46:28Z

@jtratner look ok to u?

jreback · 2014-01-15T02:24:19Z

doc/source/release.rst

@@ -73,6 +73,8 @@ Bug Fixes
 ~~~~~~~~~
  - Bug in Series replace with timestamp dict (:issue:`5797`)
  - read_csv/read_table now respects the `prefix` kwarg (:issue:`5732`).
+  - ``Panel.to_frame()`` no longer fails when ``major_axis`` is a


pls move to 0.13.1

jreback · 2014-01-15T02:24:44Z

@TomAugspurger this looks fine...pls move release notes to 0.13.1 and rebase...

Fixes bug that caused failure on Panel.to_frame() if major_axis was a MultiIndex (pandas-dev#5402) refactor reindexing to index.py API/REF: refactor bits to index. rename

TomAugspurger · 2014-01-15T15:27:40Z

@jreback rebased and moved to 0.13.1

BUG: Panel.to_frame() with MultiIndex major axis

jreback · 2014-01-15T16:12:54Z

thanks @TomAugspurger !

jtratner reviewed Nov 4, 2013
View reviewed changes

jreback reviewed Nov 4, 2013
View reviewed changes

TomAugspurger reviewed Nov 9, 2013
View reviewed changes

jreback reviewed Jan 3, 2014
View reviewed changes

jreback reviewed Jan 15, 2014
View reviewed changes

BUG: Panel.to_frame() with MultiIndex major axis

77a88a8

Fixes bug that caused failure on Panel.to_frame() if major_axis was a MultiIndex (pandas-dev#5402) refactor reindexing to index.py API/REF: refactor bits to index. rename

jreback added a commit that referenced this pull request Jan 15, 2014

Merge pull request #5417 from TomAugspurger/to-frame-multi

b15ba30

BUG: Panel.to_frame() with MultiIndex major axis

jreback merged commit b15ba30 into pandas-dev:master Jan 15, 2014

TomAugspurger deleted the to-frame-multi branch November 3, 2016 12:37

Uh oh!

BUG: Panel.to_frame() with MultiIndex major axis #5417

BUG: Panel.to_frame() with MultiIndex major axis #5417

Uh oh!

Conversation

TomAugspurger commented Nov 2, 2013

Uh oh!

jtratner commented Nov 2, 2013

all necessary.

Uh oh!

TomAugspurger commented Nov 4, 2013

Uh oh!

jtratner commented Nov 4, 2013

Uh oh!

jtratner Nov 4, 2013

Choose a reason for hiding this comment

Uh oh!

jreback commented Nov 4, 2013

Uh oh!

jtratner Nov 4, 2013

Choose a reason for hiding this comment

Uh oh!

ghost commented Nov 4, 2013

Uh oh!

jtratner commented Nov 4, 2013

Uh oh!

TomAugspurger commented Nov 4, 2013

Uh oh!

ghost commented Nov 4, 2013

Uh oh!

jtratner commented Nov 4, 2013

Uh oh!

jreback Nov 4, 2013

Choose a reason for hiding this comment

Uh oh!

jreback commented Nov 4, 2013

Uh oh!

TomAugspurger Nov 9, 2013

Choose a reason for hiding this comment

Uh oh!

TomAugspurger commented Nov 9, 2013

Uh oh!

TomAugspurger commented Nov 9, 2013

Uh oh!

TomAugspurger commented Dec 25, 2013

Uh oh!

TomAugspurger commented Jan 3, 2014

Uh oh!

jreback Jan 3, 2014

Choose a reason for hiding this comment

Uh oh!

TomAugspurger Jan 3, 2014

Choose a reason for hiding this comment

Uh oh!

jreback commented Jan 3, 2014

Uh oh!

jreback commented Jan 3, 2014

Uh oh!

jreback Jan 15, 2014

Choose a reason for hiding this comment

Uh oh!

jreback commented Jan 15, 2014

Uh oh!

TomAugspurger commented Jan 15, 2014

Uh oh!

jreback commented Jan 15, 2014

Uh oh!

Uh oh!