Test failures on FreeBSD 9.1

[neirbowj]

% uname -a
FreeBSD XXXX.saltant.net 9.1-STABLE FreeBSD 9.1-STABLE #0 r248078: Fri Mar  8 20:36:00 EST 2013     root@XXXX.saltant.net:/usr/obj/usr/src/sys/NIPPL  amd64
% pkg_info -xE pandas
py27-pandas-0.11.0.r1
% pkg_info -xE ^python
python27-2.7.3_6
% pkg_info -xE numpy
py27-numpy-1.6.2_1,1
% nosetests pandas.tests.test_index:TestMultiIndex.test_legacy_pickle
E
======================================================================
ERROR: test_legacy_pickle (pandas.tests.test_index.TestMultiIndex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/tests/test_index.py", line 1060, in test_legacy_pickle
    obj = pickle.load(open(ppath, 'r'))
  File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/local/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/local/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/local/lib/python2.7/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named multiarray

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)

[jreback]

are there other failures?
what does
test_fast.sh return?

[neirbowj]

I installed from the rc1 tarball, so I don't have test_fast.sh handy. Give me a moment to bring up a dev't environment and I'll get back to you about test_fast. In the mean time, here's the complete output from nose.

% nosetests --exe pandas

** (process:86069): WARNING **: Trying to register gtype 'GMountMountFlags' as enum when in fact it is of type 'GFlags'

** (process:86069): WARNING **: Trying to register gtype 'GDriveStartFlags' as enum when in fact it is of type 'GFlags'

** (process:86069): WARNING **: Trying to register gtype 'GSocketMsgFlags' as enum when in fact it is of type 'GFlags'
..........................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.........................................................................................................................................................................................................................................................................................SSSSSSSSSSS..........SS...................................................................................................................................................................................S..................S..S.SSSSSSS......................................S.....SSSSSSSSSSSSSSSSSSSSSSSS..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S.S....................................................E................................................................................................................................................................................................................................SSS.............................................................................................................................................................................................................................S.........................S.......................................................................................................................................................................................................................................................................................................................................E..............................................................................................................................................................S.............................................................................................................................................................S..................S...S............SSS.........S............SS...........SS.....SSSS.......................................................................................................................S......................................................................................................E.......................................S............................................................................................................
======================================================================
ERROR: test_to_string_repr_unicode (pandas.tests.test_format.TestDataFrameFormatting)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/tests/test_format.py", line 237, in test_to_string_repr_unicode
    rs = repr(ser).split('\n')
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1138, in __repr__
    return str(self)
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1097, in __str__
    return self.__bytes__()
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1107, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1125, in __unicode__
    dtype=True)
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1214, in _get_repr
    result = formatter.to_string()
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/format.py", line 146, in to_string
    idx = k.ljust(pad_space + _encode_diff(k))
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/format.py", line 168, in _encode_diff
    return len(x) - len(x.decode(encoding))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 0: ordinal not in range(128)

======================================================================
ERROR: test_legacy_pickle (pandas.tests.test_index.TestMultiIndex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/tests/test_index.py", line 1060, in test_legacy_pickle
    obj = pickle.load(open(ppath, 'r'))
  File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/local/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/local/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/local/lib/python2.7/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named multiarray

======================================================================
ERROR: http://docs.python.org/py3k/reference/datamodel.html#object.__repr__
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/tests/test_series.py", line 1361, in test_repr_should_return_str
    self.assertTrue(type(df.__repr__() == str))  # both py2 / 3
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1138, in __repr__
    return str(self)
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1097, in __str__
    return self.__bytes__()
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1107, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1125, in __unicode__
    dtype=True)
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1214, in _get_repr
    result = formatter.to_string()
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/format.py", line 146, in to_string
    idx = k.ljust(pad_space + _encode_diff(k))
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/format.py", line 168, in _encode_diff
    return len(x) - len(x.decode(encoding))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 0: ordinal not in range(128)

----------------------------------------------------------------------
Ran 3125 tests in 297.710s

FAILED (SKIP=115, errors=3)

I'm looking into the GFlags warnings now. TestDataFrameFormatting.test_eng_float_formatter generates them in the final call to fmt.reset_printoptions()

I'm also working on a minimal failing test for TestDataFrameFormatting.test_to_string_repr_unicode, because:

% nosetests pandas.tests.test_format:TestDataFrameFormatting
.
** (process:86110): WARNING **: Trying to register gtype 'GMountMountFlags' as enum when in fact it is of type 'GFlags'

** (process:86110): WARNING **: Trying to register gtype 'GDriveStartFlags' as enum when in fact it is of type 'GFlags'

** (process:86110): WARNING **: Trying to register gtype 'GSocketMsgFlags' as enum when in fact it is of type 'GFlags'
...............................................E................
======================================================================
ERROR: test_to_string_repr_unicode (pandas.tests.test_format.TestDataFrameFormatting)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/tests/test_format.py", line 237, in test_to_string_repr_unicode
    rs = repr(ser).split('\n')
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1138, in __repr__
    return str(self)
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1097, in __str__
    return self.__bytes__()
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1107, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1125, in __unicode__
    dtype=True)
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/series.py", line 1214, in _get_repr
    result = formatter.to_string()
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/format.py", line 146, in to_string
    idx = k.ljust(pad_space + _encode_diff(k))
  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/format.py", line 168, in _encode_diff
    return len(x) - len(x.decode(encoding))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 0: ordinal not in range(128)

----------------------------------------------------------------------
Ran 65 tests in 3.022s

FAILED (errors=1)

But...

% nosetests pandas.tests.test_format:TestDataFrameFormatting.test_to_string_repr_unicode
.
----------------------------------------------------------------------
Ran 1 test in 0.024s

OK

test_to_string_repr_unicode uses np.random.randn to generate test data, so pass/fail depends on earlier consumption of random numbers.

[neirbowj]

OK, when I

git clone --branch v0.11.0rc1 https://github.com/pydata/pandas pandas-0.11.0rc1
cd pandas-0.11.0rc1
python setup.py build_ext --inplace
./test_fast.sh

I get the warnings, the failure in pandas.tests.test_format:TestDataFrameFormatting.test_to_string_repr_unicode and the associated http://docs.python.org/py3k/reference/datamodel.html#object.__repr__ error, but pandas.tests.test_index:TestMultiIndex.test_legacy_pickle apparently passes.

I don't know what to do next, but will gladly take direction and respond to requests.

Please pull the latest master (I pushed some changes today), and post the output ofci/print_versions.py.

also, what is the value of pandas.options.display.encoding?

edit: Tha test name above is now fixed in master.

[neirbowj]

% git describe
v0.11.0rc1-27-g9c05da7
% ci/print_versions.py

INSTALLED VERSIONS
------------------
Python: 2.7.3.final.0
OS: FreeBSD 9.1-STABLE FreeBSD 9.1-STABLE #0 r248078: Fri Mar  8 20:36:00 EST 2013     root@XXXX.saltant.net:/usr/obj/usr/src/sys/NIPPL amd64

Cython: 0.17.1
Numpy: 1.6.2
Scipy: 0.11.0
statsmodels: Not installed
    patsy: Not installed
scikits.timeseries: Not installed
dateutil: 2.1
pytz: 2013b
PyTables: Not Installed
    numexpr: Not Installed
matplotlib: 1.2.0
openpyxl: Not installed
xlrd: Not installed
xlwt: Not installed
sqlalchemy: Not installed

And furthermore...

>>> import pandas
/usr/local/lib/python2.7/site-packages/pytz-2013b-py2.7.egg/pytz/__init__.py:35: UserWarning: Module pandas was already imported from pandas/__init__.pyc, but /usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg is being added to sys.path
>>> pandas.options.display.encoding
'US-ASCII'
>>>

Your terminal doesn't support utf-8/unicode, or at least doesn't report it in a way pandas
can discern (the code is in core/format.py:detect_console_encoding()).

Obviously, the test needs to fail more gracefuly in such a situation, but really it's signalling
that while pandas supports unicode, your environment does not.

If you're only working with ASCII data, your should be fine in principle btw.

But there's something else going on here.

  File "/usr/local/lib/python2.7/site-packages/pandas-0.11.0rc1-py2.7-freebsd-9.1-STABLE-amd64.egg/pandas/core/format.py", line 168, in _encode_diff
    return len(x) - len(x.decode(encoding))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 0: ordinal not in range(128)

That should have been a UnicodeEncodeError (which would have been caught and the test passed).

I expected to see:

In [7]: x=u'\u03c3'
   ...: len(x)-len(x.decode('ascii'))
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-7-33671dd44d61> in <module>()
      1 x=u'\u03c3'
----> 2 len(x)-len(x.decode('ascii'))

UnicodeEncodeError: 'ascii' codec can't encode character u'\u03c3' in position 0: ordinal not in range(128)

x is a unicode string, which gets implicitly encoded into ASCII when decoded,
it doesn't make sense that that would succeed and the decode immediately
following would fail.

Can you break in with pdb and print out x at the point of failure?

nosetests pandas/tests/test_fomat -m repr --pdb
<break in>
(pdb) print x

also, what are the values of the $LC_ALL and $LANG envars?

[neirbowj]

% nosetests pandas/tests/test_format.py -m repr --pdb
.
** (process:2949): WARNING **: Trying to register gtype 'GMountMountFlags' as enum when in fact it is of type 'GFlags'

** (process:2949): WARNING **: Trying to register gtype 'GDriveStartFlags' as enum when in fact it is of type 'GFlags'

** (process:2949): WARNING **: Trying to register gtype 'GSocketMsgFlags' as enum when in fact it is of type 'GFlags'
.............> /home/obrienjw/src/github/pandas/pandas/core/format.py(168)_encode_diff()
-> return len(x) - len(x.decode(encoding))
(Pdb) print x
*** UnicodeEncodeError: 'ascii' codec can't encode character u'\u03c3' in position 0: ordinal not in range(128)
(Pdb)

Neither $LC_ALL nor $LANG are defined in the environment.

This has been vexing me but I just breathed a sigh of relief since you just pointed out the UnicodeEncodeError vs. UnicodeDecodeError difference. I hadn't noticed it, and was racking my brain to understand why the except block wasn't catching it. D'oh! At least now there is an explanation for why this test passes sometimes and fails other times (i.e. different exceptions).

As for the UTF-8 support, I'll see about turning on support in my terminal so that I can run tests in a richer environment. I'm the pandas maintainer for FreeBSD so it's not just my data I need to worry about.

I'm pretty sure the code that's misbehaving (although the wrong exception is puzzling)
can actually be removed (#3364), it was a stopgap that's no longer necessary after other
unicode-related changes. But that's not a change I'll merge a few days before a major release.

I can't repro any of this on my box, but if you figure out why the DecodeError is happening
instead of the EncodeError that's 99% of it.

was the pickle issue a false alarm?

[neirbowj]

Oops. Not sure about the pickle test. Will see if I can reproduce it tonight.

[jreback]

fyi....do a clean git clone and try.....the error message on there, can't find multiarray is a numpy module (so also make sure that it is loading correct numpy), etc.

Relavent perhaps:
http://stackoverflow.com/questions/9641916/python-pandas-cant-find-numpy-core-multiarray-when-importing-pandas

might you be using a 32bit compiled numpy on an amd64 system?