test_constructor_compound_dtypes and test_invalid_index_types fail in parse_datetime_string

[yarikoptic]

Debian stretch/sid, cython 0.22.1-2, pandas v0.17.0rc1-92-gc6bcc99

$> locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

nosetest pandas leads to

$> nosetests -s -v --failed pandas --pdb
#1982 test_invalid_index_types (pandas.tseries.tests.test_frequencies.TestFrequencyInference) ... > /usr/lib/python2.7/dist-packages/dateutil/parser.py(310)parse()
-> res, skipped_tokens = self._parse(timestr, **kwargs)
(Pdb) p timestr
u'\u05e4\u05d0\u05e3\u05df\u05e1\u05da\u05e8\u05e1\u05e7\u05df'
(Pdb) self._parse(timestr, **kwargs)
(Pdb) up
> /usr/lib/python2.7/dist-packages/dateutil/parser.py(748)parse()
-> return DEFAULTPARSER.parse(timestr, **kwargs)
(Pdb) p timestr
u'\u05e4\u05d0\u05e3\u05df\u05e1\u05da\u05e8\u05e1\u05e7\u05df'
(Pdb) p DEFAULTPARSER
<dateutil.parser.parser object at 0x7f79dcacf090>
*(Pdb) p timestr.encode()
*** UnicodeEncodeError: UnicodeEncodeError('ascii', u'\u05e4\u05d0\u05e3\u05df\u05e1\u05da\u05e8\u05e1\u05e7\u05df', 0, 10, 'ordinal not in range(128)')
*(Pdb) p timestr.encode('utf-8')
'\xd7\xa4\xd7\x90\xd7\xa3\xd7\x9f\xd7\xa1\xd7\x9a\xd7\xa8\xd7\xa1\xd7\xa7\xd7\x9f'
(Pdb) 
ERROR
#3617 test_constructor_compound_dtypes (pandas.tests.test_frame.TestDataFrame) ... > /usr/lib/python2.7/dist-packages/dateutil/parser.py(310)parse()
-> res, skipped_tokens = self._parse(timestr, **kwargs)
(Pdb) 
ERROR

======================================================================
ERROR: test_invalid_index_types (pandas.tseries.tests.test_frequencies.TestFrequencyInference)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tests/test_frequencies.py", line 565, in test_invalid_index_types
    self.assertRaises(ValueError, lambda : frequencies.infer_freq(i))
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/testing.py", line 1880, in assertRaises
    _callable(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tests/test_frequencies.py", line 565, in <lambda>
    self.assertRaises(ValueError, lambda : frequencies.infer_freq(i))
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/frequencies.py", line 855, in infer_freq
    index = pd.DatetimeIndex(index)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/decorators.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/index.py", line 315, in __new__
    subarr = tools.to_datetime(data, box=False)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/decorators.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 276, in to_datetime
    unit=unit, infer_datetime_format=infer_datetime_format)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 395, in _to_datetime
    return _convert_listlike(arg, box, format)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 372, in _convert_listlike
    require_iso8601=require_iso8601)
  File "pandas/tslib.pyx", line 1828, in pandas.tslib.array_to_datetime (pandas/tslib.c:34125)
    cpdef array_to_datetime(ndarray[object] values, errors='raise',
  File "pandas/tslib.pyx", line 2017, in pandas.tslib.array_to_datetime (pandas/tslib.c:33760)
    raise
  File "pandas/tslib.pyx", line 2011, in pandas.tslib.array_to_datetime (pandas/tslib.c:33658)
    oresult[i] = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1602, in pandas.tslib.parse_datetime_string (pandas/tslib.c:27245)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/usr/lib/python2.7/dist-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/lib/python2.7/dist-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

======================================================================
ERROR: test_constructor_compound_dtypes (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_frame.py", line 14395, in test_constructor_compound_dtypes
    f('M8[ns]')
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_frame.py", line 14384, in f
    columns=["A", "B", "C"], dtype=dtype)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 264, in __init__
    arrays, columns = _to_arrays(data, columns, dtype=dtype)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 5216, in _to_arrays
    dtype=dtype)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 5299, in _list_to_arrays
    coerce_float=coerce_float)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 5366, in _convert_object_array
    arrays = [ convert(arr) for arr in content ]
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 5363, in convert
    arr = com._possibly_cast_to_datetime(arr, dtype)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/common.py", line 1998, in _possibly_cast_to_datetime
    value = to_datetime(value, errors=errors)._values
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/decorators.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 276, in to_datetime
    unit=unit, infer_datetime_format=infer_datetime_format)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 395, in _to_datetime
    return _convert_listlike(arg, box, format)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 372, in _convert_listlike
    require_iso8601=require_iso8601)
  File "pandas/tslib.pyx", line 1828, in pandas.tslib.array_to_datetime (pandas/tslib.c:34125)
    cpdef array_to_datetime(ndarray[object] values, errors='raise',
  File "pandas/tslib.pyx", line 2017, in pandas.tslib.array_to_datetime (pandas/tslib.c:33760)
    raise
  File "pandas/tslib.pyx", line 2011, in pandas.tslib.array_to_datetime (pandas/tslib.c:33658)
    oresult[i] = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1602, in pandas.tslib.parse_datetime_string (pandas/tslib.c:27245)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/usr/lib/python2.7/dist-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/lib/python2.7/dist-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

----------------------------------------------------------------------
Ran 2 tests in 59.805s

[jreback]

yeh, the we these on windows because its actually not giving us the correct error (its in a weird encoding).

I never could figure out why this works on linux/osx but not elsewhere.

@yarikoptic maybe the same issue.

@sinhrks

[yarikoptic]

The same as ... ? in my case it is as non-windows as it gets (Debian GNU/Linux ;))

[jreback]

what I mean is that we skip these on windows (never could really figure it out), yet it works on our travis builds (linux) and macosx. So something is different. I think its some kind of encoding issue.

[FRidh]

Both also fail with Nix. See #11287.