Taking mean on dataframe with uint16 gives wrong number

[kyleco]

See this very strange behavior when taking the mean of a dataframe that has a series with uint16 datatype. Why does the mean of column y change after including column x?

Python 2.7.7 |Anaconda 2.0.1 (x86_64)| (default, Jun  2 2014, 12:48:16)
Type "copyright", "credits" or "license" for more information.

IPython 2.1.0 -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: import random

In [4]:

In [4]: np.__version__
Out[4]: '1.8.1'

In [5]: pd.__version__
Out[5]: '0.14.0'

In [6]:

In [6]: y = np.array([random.randint(1900,2000) for x in range(0,2000)])

In [7]: y.mean()
Out[7]: 1950.0115000000001

In [8]: y.astype(np.uint16).mean()
Out[8]: 1950.0115000000001

In [9]:

In [9]: d1 = pd.DataFrame()

In [10]: d1['y'] = y.astype(np.uint16)

In [11]: d1.mean()
Out[11]:
y    16.6995
dtype: float64

In [12]:

In [12]: d1['x'] = y.astype(np.int16)

In [13]: d1.mean()
Out[13]:
y    1950.0115
x    1950.0115
dtype: float64

[jreback]

can u post

pd.show_versions()?

[kyleco]

In [14]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.7.final.0
python-bits: 64
OS: Darwin
OS-release: 13.3.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.14.0
nose: 1.3.3
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.14.0
statsmodels: 0.6.0.dev-2e806fc
IPython: 2.1.0
sphinx: 1.2.2
patsy: 0.2.1
scikits.timeseries: None
dateutil: 2.2
pytz: 2014.3
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.3.1
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.5.5
lxml: 3.3.5
bs4: 4.3.1
html5lib: None
bq: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.4
pymysql: None
psycopg2: None

[jreback]

seems related this this: #6954

though since u r 64 bit the accumulators should be 64 bit

if u want to debug would be helpful

step into the function and see if u can figure what's going on

[kyleco]

Thx. Really pressed for time at the moment. Will try to take a look later.

[dsm054]

Okay, I can reproduce:

>>> df = pd.DataFrame()
>>> y = np.random.randint(1900, 2000, 2000).astype(np.uint16)
>>> y.mean()
1949.4085
>>> df["y"] = y
>>> df.mean()
y    16.0965
dtype: float64
>>> df["x"] = y
>>> df.mean()
y    16.0965
x    16.0965
dtype: float64
>>> df["x"] = y.astype(np.int16)
>>> df.mean()
y    1949.4085
x    1949.4085
dtype: float64

with

>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.0.final.0
python-bits: 32
OS: Linux
OS-release: 3.13.0-24-generic
machine: i686
processor: i686
byteorder: little
LC_ALL: None
LANG: en_CA.UTF-8

pandas: 0.13.1-836-g3d2c80f
nose: 1.3.1
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.15.0.dev-dafe24c
statsmodels: 0.6.0.dev-e1be28e
IPython: 1.2.1
sphinx: None
patsy: 0.2.1
scikits.timeseries: None
dateutil: 2.2
pytz: 2013.9
bottleneck: None
tables: None
numexpr: 2.3.1
matplotlib: 1.4.x
openpyxl: 1.8.4
xlrd: 0.9.2
xlwt: 0.8.0
xlsxwriter: 0.5.3
lxml: 3.3.3
bs4: 4.3.2
html5lib: 0.999
bq: None
apiclient: None
rpy2: None
sqlalchemy: None
pymysql: None
psycopg2: None

but note the version number (pre-14). So far that's the only place I've succeeded.

[jreback]

@dsm054 that's an older version of pandas yes?

[dsm054]

Yep. I haven't managed to repro on trunk, but that's because the worker nodes have rather random versions.. testing more modern stuff as we speak.

[dsm054]

Wasn't able to reproduce on anything more recent, and my efforts to bisect to confirm that @jreback's earlier patch fixed the problem were stymied by an unexpected AttributeError: 'module' object has no attribute 'MS_RESO' error, which apparently has happened to at least one other person. Odd, because the Cython file tslib.pyx is the same but the resulting C doesn't include MS_RESO. Will try to sort that out later.

[kyleco]

Reproduced on another machine (but running same versions of pandas and numpy)

Python 2.7.5 |Anaconda 2.0.1 (64-bit)| (default, Jul  1 2013, 12:37:52) [MSC v.1500 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 2.1.0 -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: import random

In [4]:

In [4]: np.__version__
Out[4]: '1.8.1'

In [5]: pd.__version__
Out[5]: '0.14.0'

In [6]:

In [6]: y = np.array([random.randint(1900,2000) for x in range(0,2000)])

In [7]: y.mean()
Out[7]: 1949.6875

In [8]: y.astype(np.uint16).mean()
Out[8]: 1949.6875

In [9]:

In [9]: d1 = pd.DataFrame()

In [10]: d1['y'] = y.astype(np.uint16)

In [11]: d1.mean()
Out[11]:
y    16.3755
dtype: float64

In [12]:

In [12]: d1['x'] = y.astype(np.int16)

In [13]: d1.mean()
Out[13]:
y    1949.6875
x    1949.6875
dtype: float64

In [14]:

In [14]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.5.final.0
python-bits: 64
OS: Windows
OS-release: 8
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.14.0
nose: 1.3.3
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.14.0
statsmodels: 0.6.0.dev-2e806fc
IPython: 2.1.0
sphinx: 1.2.2
patsy: 0.3.0
scikits.timeseries: None
dateutil: 1.5
pytz: 2014.3
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.3.1
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.5.5
lxml: 3.3.5
bs4: 4.3.1
html5lib: 0.95-dev
bq: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.4
pymysql: None
psycopg2: None

[jreback]

@kcarlson00 I believe this is fixed in 0.14.1 AFAICT. (see that references issue).

Essentially the issue is that the accumulator in mean/sum is the same dtype as the input, rather than being upcast to say int64 (which can hold an overflow). IMHO this is a numpy issue, but they push it back on the user (pandas). So that's why we now do just that, automatically upcast the accumular to avoid these issues.

If you (or anyone) else can reprodu with 0.14.1, pls reopen.

[MattDMo]

@jreback so what was happening was the accumulator used whilst summing the values during the process of taking the mean, overflowed because it was an unsigned 16-bit int (np.uint16), but changing it to int16 gave it enough space?

[jreback]

no I think the accumulators that numpy picks for units might be incorrect (a guess)
the overflow is usually (but not only the reason)

u can experiment by doing something like

numpy_array.mean(dtype=uint16)

[kyleco]

I upgraded to 0.14.1 and the issue is resolved. Thanks.

On Sun, Aug 10, 2014 at 9:39 AM, jreback notifications@github.com wrote:

no I think the accumulators that numpy picks for units might be incorrect
(a guess)
the overflow is usually (but not only the reason)

u can experiment by doing something like

numpy_array.mean(dtype=uint16)

—
Reply to this email directly or view it on GitHub
#7976 (comment).