pandas appears to crash doing pivot table from 10 minute examples

[rmschne]

pandas 0.11.0 installed with Python 2.7.2 using "pip install pandas" on fully patched mac osx 10.8.5, running in a virtual environment. running any pivot table against any DataFrame create so for -- nothing happens. pressing ctrl-c I see the latest lines run are:

422         while com._long_prod(sizes) >= 2 ** 63:
423             i = len(sizes)

--> 424 while com._long_prod(sizes[:i]) >= 2 ** 63:
425 i -= 1

The problem is reproducible using http://pandas.pydata.org/pandas-docs/stable/10min.html and running:

from pandas import DataFrame, Series
import pandas as pd
import numpy as np

then running lines [97], [98], and [99] in iPython and Python native.

I can't see any way to debug or go further.

[cpcloud]

can u put up the exact steps u used to create the virtualenv and build pandas?

[rmschne]

to build pandas, I used the command "pip install pandas".

To install the virtualenv, it was so long ago I cannot remember. I just followed the simple instructions they provide. To create the virtualenv i run the command "source /users/rmschne/.virtualenv/rmspython/bin/activate".

Is there a "right" way for pandas to work? I did not notice anything special in docs.

Update: I looked in my records, and I used http://docs.python-guide.org/en/latest/starting/install/osx.html for how to install virtualenv, pip, etc.

[cpcloud]

Did you install Cython? I don't think pandas installs it by default.

[rmschne]

I did not install Cython. per http://pandas.pydata.org/pandas-docs/stable/install.html#dependencies it is "Only necessary to build development version. Version 0.17.1 or higher".

However, I just installed it with "pip install cython" and it appeared to be successful.

then, running:

import pandas as pd
import numpy as np
dates = pd.date_range('20130101',periods=6)
df=pd.DataFrame(np.random.randn(6,4),index=dates,columns=list('ABCD'))
pd.pivot_table(df,values='D',rows=['A','B'], cols=['C'])

The session hangs. No error message. ctrl-c returns:

File "", line 1, in
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/tools/pivot.py", line 106, in pivot_table
table = agged.unstack(to_unstack)
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/core/frame.py", line 3971, in unstack
return unstack(self, level)
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/core/reshape.py", line 365, in unstack
return _unstack_multiple(obj, level)
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/core/reshape.py", line 266, in _unstack_multiple
unstacked = dummy.unstack('placeholder')
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/core/frame.py", line 3971, in unstack
return unstack(self, level)
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/core/reshape.py", line 369, in unstack
return _unstack_frame(obj, level)
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/core/reshape.py", line 407, in _unstack_frame
value_columns=obj.columns)
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/core/reshape.py", line 76, in init
self._make_sorted_values_labels()
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/core/reshape.py", line 87, in _make_sorted_values_labels
comp_index, obs_ids = get_compressed_ids(to_sort, sizes)
File "/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/core/reshape.py", line 425, in get_compressed_ids
i -= 1

[cpcloud]

you're right you don't need it my bad

[cpcloud]

here's what you should do, if you're up for it, since i can't reproduce this on the latest master or on pandas from PyPI:

1. install ipython if you haven't already
2. put the above example into a file
3. start up ipython and type run -d the_file_where_you_saved_the_above_example.py
4. step through the example using some combination of s (step) and n (next) until you find the point at which the hang occurs.
5. report back with one or some of the location/filename/line number where it hangs.

[rmschne]

hang is here:

ipdb> n

/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/tools/pivot.py(103)pivot_table()
102 table = agged
--> 103 if table.index.nlevels > 1:
104 to_unstack = [agged.index.names[i]

ipdb> n

/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/tools/pivot.py(104)pivot_table()
103 if table.index.nlevels > 1:
--> 104 to_unstack = [agged.index.names[i]
105 for i in range(len(rows), len(keys))]

ipdb> n

/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/tools/pivot.py(105)pivot_table()
104 to_unstack = [agged.index.names[i]
--> 105 for i in range(len(rows), len(keys))]
106 table = agged.unstack(to_unstack)

ipdb> n

/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/tools/pivot.py(105)pivot_table()
104 to_unstack = [agged.index.names[i]
--> 105 for i in range(len(rows), len(keys))]
106 table = agged.unstack(to_unstack)

ipdb> n

/Users/rmschne/.virtualenv/rmspython/lib/python2.7/site-packages/pandas/tools/pivot.py(106)pivot_table()
105 for i in range(len(rows), len(keys))]
--> 106 table = agged.unstack(to_unstack)
107

ipdb>

HANG FROM HERE

[cpcloud]

can u try just a simple

In [1]: df = DataFrame(randn(60000, 4))

In [2]: df.unstack()
Out[2]:
0  0    -0.395
   1    -0.633
   2     0.651
   3    -2.116
   4    -0.687
   5     0.350
   6    -0.445
   7     0.262
   8     1.306
   9    -1.451
   10   -0.312
   11   -0.676
   12   -0.035
   13   -1.226
   14    0.119
...
3  59985   -2.490
   59986    0.891
   59987   -0.081
   59988   -0.180
   59989   -0.767
   59990    1.075
   59991    0.301
   59992   -0.627
   59993    2.084
   59994    0.221
   59995   -1.321
   59996   -0.341
   59997   -0.843
   59998   -0.121
   59999    0.238
Length: 240000, dtype: float64

[rmschne]

thanks for staying with me. got same:

import pandas as pd
import numpy as np
df=pd.DataFrame(np.random.randn(60000,4))
df.unstack()
0 0 0.564648
1 -1.859481
2 -0.082655
3 -0.349804
4 -0.180945
5 1.545953
6 -1.253212
7 0.283621
8 -1.554463
9 0.677267
10 -0.154183
11 -0.765514
12 -0.271065
13 -1.162593
14 0.099856
...
3 59985 -0.314345
59986 -1.854728
59987 -1.411842
59988 0.837962
59989 1.780513
59990 -1.535803
59991 0.064691
59992 0.326311
59993 -1.609834
59994 0.919482
59995 0.499292
59996 1.141393
59997 -0.440700
59998 -0.755849
59999 1.166247
Length: 240000, dtype: float64

[cpcloud]

ok so now try df.unstack(0)...does that work?

[cpcloud]

few more things

1. build the latest git master and see if the problem is alleviated
2. if the issue exists after that it might be something with libraries specific to mac

so far git master builds and runs on windows (i think 7 is what ppl are building with @jreback?), arch linux, ubuntu 12.04 and 12.10, fedora 17, centos and a couple of mac versions not sure which ones though. personally i've tested all the linux versions i just mentioend except centos and they're fine.

can u post the output of agged and of to_unstack (at the point right before it hangs)?

[michaelaye]

10.8.5? Are you running a development version of OSX?

[rmschne]

Re michaelaye's question. No. Macbook Pro OSX Ver 10.8.4.
Re cpcloud's requests. Will do shortly.

[rmschne]

df.unstack(0) works as expected.

[rmschne]

@michaelaye ... following are all the versions in place in the virtual env. While OSX is not development, is the comment from yolk -l "active development" for Python 2.7.2 relevant to your question? I never noticed word "development". I've not touched the built-in Python to my knowledge. New machine in Nov 2012.

Cython - 0.19.1 - active
Django - 1.5.1 - active
MySQL-python - 1.2.4b4 - active
Python - 2.7.2 - active development (/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload)
distribute - 0.6.28 - active
django-ical - 1.1 - active
django-mailchimp - 0.1.30 - active
icalendar - 3.1 - active
ipython - 0.13.2 - active
mailsnake - 1.6.1 - active
matplotlib - 1.2.1 - active
numpy - 1.7.1 - active
pandas - 0.11.0 - active
pip - 1.2.1 - active
python-dateutil - 2.1 - active
pytz - 2013b - active
requests - 0.14.2 - active
scipy - 0.11.0 - active
setuptools - 0.6c11 - active
simplejson - 2.6.2 - active
six - 1.3.0 - active
wsgiref - 0.1.2 - active development (/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7)
yolk - 0.4.3 - active