BUG: HDFStore TypeError: decoding Unicode is not supported

[TomAugspurger]

On latest master 0.12.0-315-ga99b3be, when I try to select a table from an HDFStore, I get a Unicode Error

In [4]: df = store.select('monthly/data/m1998_01')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-67c3fabe2905> in <module>()
----> 1 df = store.select('monthly/data/m1998_01')

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in select(self, key, where, start, stop, columns, iterator, chunksize, auto_close, **kwargs)
    579         where = _ensure_term(where)
    580         s = self._create_storer(group)
--> 581         s.infer_axes()
    582 
    583         # what we are actually going to do for a chunk

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in infer_axes(self)
   2001         if s is None:
   2002             return False
-> 2003         self.get_attrs()
   2004         return True
   2005 

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in get_attrs(self)
   2740         t = self.table
   2741         self.index_axes = [a.infer(t)
-> 2742                                          for a in self.indexables if a.is_an_indexable]
   2743         self.values_axes = [a.infer(t)
   2744                                          for a in self.indexables if not a.is_an_indexable]

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in indexables(self)
   2793 
   2794             self._indexables.extend(
-> 2795                 [f(i, c) for i, c in enumerate(self.attrs.values_cols)])
   2796 
   2797         return self._indexables

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in f(i, c)
   2790                 if c in dc:
   2791                     klass = DataIndexableCol
-> 2792                 return klass.create_for_block(i=i, name=c, pos=base_pos + i, version=self.version)
   2793 
   2794             self._indexables.extend(

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in create_for_block(cls, i, name, cname, version, **kwargs)
   1537             pass
   1538 
-> 1539         return cls(name=name, cname=cname, **kwargs)
   1540 
   1541     def __init__(self, values=None, kind=None, typ=None,

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/io/pytables.py in __init__(self, values, kind, typ, cname, data, block, **kwargs)
   1544             values=values, kind=kind, typ=typ, cname=cname, **kwargs)
   1545         self.dtype = None
-> 1546         self.dtype_attr = u("%s_dtype" % self.name)
   1547         self.set_data(data)
   1548 

/Users/tom/Envs/statsmodels/lib/python2.7/site-packages/pandas/pandas/compat/__init__.py in u(s)
    189 
    190     def u(s):
--> 191         return unicode(s, "unicode_escape")
    192 
    193 

TypeError: decoding Unicode is not supported

I'm still trying to whittle down where this behavior changed, but at least as of a99b3be, it worked. I'm going to go through commits changing HDFStore, but maybe someone knows exactly where this might be from?

I'll also try to come up with a minimal example that reproduces. Simple ones aren't doing it which is a good thing I guess.

[jreback]

what is in this file? can you post it?

[jreback]

are you using pytables 3.0.0?

and did you create it with a different version?

[TomAugspurger]

I've selected out a column and uploaded it here: https://www.dropbox.com/s/mjaomv981ue4cu6/bug.h5 (98 MB)

I'm on tables 3.0

Works fine as of a2d8b3d, so its pretty recent.

[jreback]

well..just merged in eval, (which changed pytables parsing)...could be a bug....

[TomAugspurger]

Seems to be here: error as of 50502fb CLN/DOC: remove __getitem__ from docs, good as of ba9c143 Merge pull request #4852 from jreback/seq2.

I'll see if I can trim down that example. 98 MB isn't exactly minimal :)

[jreback]

hmmm....its a trivial fix....hold a sec

[jreback]

@TomAugspurger I put up a PR #4856 which allows this to work, but still not sure why this would be happending (esp since the commit you point too just changes docs and nothing related to this)

can you create your hdf with current master (w/o this PR) and see if it reproduces?

I assume you created it before?

[TomAugspurger]

Yes I created it before (early in the summer).

Your PR fixes the unicode error.

With an HDFStore created on the current master, same data just round-tripped through csv.

In [1]: print pd.__version__
0.12.0-315-ga99b3be

In [2]: df = pd.read_csv('bug.csv', index_col=[0,1,2])

In [3]: store_master = pd.HDFStore('master.h5')

In [4]: store_master.appe
store_master.append              store_master.append_to_multiple  

In [4]: store_master.append('df', df)

In [5]: store_master.sele
store_master.select                 store_master.select_as_multiple     
store_master.select_as_coordinates  store_master.select_column          

In [5]: store_master.select('df')

is all good.

This is very odd considering the changes in that commit :/

So to sum up new tables (i.e. created on current master) seem to be fine with this data. Old tables broke somewhere in the eval merge, but your PR fixes it.

[jreback]

very odd....guess I'll merge it its pretty non-invasive

[TomAugspurger]

Thanks.