test_pytables failures on Windows 64-bit, PyTables 2.4.0

[wesm]

C:\code\pandas>c:\python27\scripts\nosetests.exe pandas\io\tests\test_pytables.py
.S......SSS...........F.F...S..S...E....F..E..E.....E........E........E....
======================================================================
ERROR: test_put (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 179, in test_put
    ValueError, self.store.put, 'f', df[10:], append=True)
  File "c:\python27\lib\unittest\case.py", line 471, in assertRaises
    callableObj(*args, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 447, in put
    self._write_to_group(key, value, table=table, append=append, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 664, in _write_to_group
    wrapper(value)
  File "C:\code\pandas\pandas\io\pytables.py", line 652, in <lambda>
    complib=complib, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 808, in _write_frame_table
    t.write(axes=axes, obj=df, append=append, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 2145, in write
    min_itemsize=min_itemsize, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 1896, in create_axes
    self.validate(existing_table)
  File "C:\code\pandas\pandas\io\pytables.py", line 1532, in validate
    % (c, getattr(self, c, None), getattr(other, c, None)))
Exception: invalid combinate of [non_index_axes] on appending data [[(1, ['A', 'B', 'C', 'D'])]] vs current table [[(1, ['A', 'B', 'C', 'D', 'string', 'string2'])]]

======================================================================
ERROR: test_reopen_handle (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 142, in test_reopen_handle
    self.store.open('w', warn=False)
  File "C:\code\pandas\pandas\io\pytables.py", line 291, in open
    self.handle = _tables().openFile(self.path, self.mode)
  File "c:\python27\lib\site-packages\tables\file.py", line 230, in openFile
    return File(filename, mode, title, rootUEP, filters, **kwargs)
  File "c:\python27\lib\site-packages\tables\file.py", line 495, in __init__
    self._g_new(filename, mode, **params)
  File "hdf5Extension.pyx", line 317, in tables.hdf5Extension.File._g_new (tables\hdf5Extension.c:3039)
HDF5ExtError: HDF5 error back trace

  File "..\..\hdf5-1.8.8\src\H5F.c", line 1440, in H5Fcreate
    unable to create file
  File "..\..\hdf5-1.8.8\src\H5F.c", line 1243, in H5F_open
    unable to truncate a file which is already open

End of HDF5 error back trace

Unable to open/create file '__test__.h5'

======================================================================
ERROR: test_select_as_multiple (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 1592, in test_select_as_multiple
    self.store.append('df2', df2)
  File "C:\code\pandas\pandas\io\pytables.py", line 512, in append
    self._write_to_group(key, value, table=True, append=True, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 664, in _write_to_group
    wrapper(value)
  File "C:\code\pandas\pandas\io\pytables.py", line 652, in <lambda>
    complib=complib, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 808, in _write_frame_table
    t.write(axes=axes, obj=df, append=append, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 2145, in write
    min_itemsize=min_itemsize, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 1891, in create_axes
    raise Exception("cannot find the correct atom type -> [dtype->%s] %s" % (b.dtype.name, str(detail)))
Exception: cannot find the correct atom type -> [dtype->object] list index out of range

======================================================================
ERROR: test_start_stop (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 1631, in test_start_stop
    self.store.append('df', df)
  File "C:\code\pandas\pandas\io\pytables.py", line 512, in append
    self._write_to_group(key, value, table=True, append=True, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 664, in _write_to_group
    wrapper(value)
  File "C:\code\pandas\pandas\io\pytables.py", line 652, in <lambda>
    complib=complib, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 808, in _write_frame_table
    t.write(axes=axes, obj=df, append=append, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 2145, in write
    min_itemsize=min_itemsize, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 1896, in create_axes
    self.validate(existing_table)
  File "C:\code\pandas\pandas\io\pytables.py", line 1532, in validate
    % (c, getattr(self, c, None), getattr(other, c, None)))
Exception: invalid combinate of [non_index_axes] on appending data [[(1, ['A', 'B'])]] vs current table [[(1, ['A', 'B', 'C', 'D'])]]

======================================================================
ERROR: test_table_values_dtypes_roundtrip (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 787, in test_table_values_dtypes_roundtrip
    self.store.append('df1', df1)
  File "C:\code\pandas\pandas\io\pytables.py", line 512, in append
    self._write_to_group(key, value, table=True, append=True, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 664, in _write_to_group
    wrapper(value)
  File "C:\code\pandas\pandas\io\pytables.py", line 652, in <lambda>
    complib=complib, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 808, in _write_frame_table
    t.write(axes=axes, obj=df, append=append, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 2145, in write
    min_itemsize=min_itemsize, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 1896, in create_axes
    self.validate(existing_table)
  File "C:\code\pandas\pandas\io\pytables.py", line 1532, in validate
    % (c, getattr(self, c, None), getattr(other, c, None)))
Exception: invalid combinate of [non_index_axes] on appending data [[(1, ['a'])]] vs current table
[[(1, ['A', 'B', 'C', 'D'])]]

======================================================================
ERROR: test_unique (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 1509, in test_unique
    self.store.append('df2', df2, data_columns=['string'])
  File "C:\code\pandas\pandas\io\pytables.py", line 512, in append
    self._write_to_group(key, value, table=True, append=True, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 664, in _write_to_group
    wrapper(value)
  File "C:\code\pandas\pandas\io\pytables.py", line 652, in <lambda>
    complib=complib, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 808, in _write_frame_table
    t.write(axes=axes, obj=df, append=append, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 2145, in write
    min_itemsize=min_itemsize, **kwargs)
  File "C:\code\pandas\pandas\io\pytables.py", line 1891, in create_axes
    raise Exception("cannot find the correct atom type -> [dtype->%s] %s" % (b.dtype.name, str(detail)))
Exception: cannot find the correct atom type -> [dtype->object] list index out of range

======================================================================
FAIL: test_get (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 160, in test_get
    self.assertRaises(KeyError, self.store.get, 'b')
AssertionError: KeyError not raised

======================================================================
FAIL: test_keys (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 67, in test_keys
    self.assertEquals(len(self.store), 5)
AssertionError: 24 != 5

======================================================================
FAIL: test_remove (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 868, in test_remove
    self.assertEquals(len(self.store), 1)
AssertionError: 27 != 1

----------------------------------------------------------------------
Ran 75 tests in 12.297s

FAILED (SKIP=6, errors=6, failures=3)

C:\code\pandas>

[jreback]

@wesm is this using current master?

[jreback]

@wesm it looks like the test file is from a previous run - sometimes when I debug I end up and the tests don't clean up the file is not deleted, next run can have weird errors...

[wesm]

I randomized the hdf5 file name between individual tests. it appears that test_append_with_data_columns somehow is creating 2 file handles to the same file

[wesm]

Current status with that commit:

C:\code\pandas>c:\python27\scripts\nosetests.exe pandas
..............................................................SS.........................................................................................................................................................................................................................................................................S....E.SSS.................S..S.....................................................SS...........................................................................................................................................................................S.....................SSSSSSS....................................c:\python27\lib\site-packages\scikits\statsmodels\tools\tools.py:256: FutureWarning: The default of `prepend` will be changed to True in the
next release, use explicit prepend
  "next release, use explicit prepend", FutureWarning)
...S............................................S...........................................................................................................................S.......................................................................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................................................................................................................S.......................................................................................................................................................................................................................................................SSS........................................................................................................................................................................................................................................S........................SSS...................................................................................................................................................................................................................................................................................................................................................................................S..........................................................................................................................................................................S...S............SSS.........S............SS...........SS.....SSSS.................................................................................................S.................................................................................................................................S.........................................................................................................
======================================================================
ERROR: test_append_with_data_columns (pandas.io.tests.test_pytables.TestHDFStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\code\pandas\pandas\io\tests\test_pytables.py", line 40, in tearDown
    os.remove(self.path)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: '__3sSGxBdLbs__.h5'

----------------------------------------------------------------------
Ran 2884 tests in 80.092s

FAILED (SKIP=47, errors=1)

[jreback]

@wesm pushed an updated....still don't know where it failed, this will allow the tests to complete...can you post: ptdump -av of the file .h5 that was last created (so I can see where it failed)...thx

[wesm]

C:\code\pandas>c:\python27\python c:\python27\Scripts\ptdump -av __3sSGxBdLbs__.h5
/ (RootGroup) ''
  /._v_attrs (AttributeSet), 4 attributes:
   [CLASS := 'GROUP',
    PYTABLES_FORMAT_VERSION := '2.0',
    TITLE := '',
    VERSION := '1.0']
/df (Group) ''
  /df._v_attrs (AttributeSet), 12 attributes:
   [CLASS := 'GROUP',
    TITLE := '',
    VERSION := '1.0',
    data_columns := ['A', 'B', 'string', 'string2'],
    index_cols := [(0, 'index')],
    levels := 1,
    nan_rep := 'nan',
    non_index_axes := [(1, ['A', 'B', 'C', 'D', 'string', 'string2'])],
    pandas_type := 'frame_table',
    pandas_version := '0.10.1',
    table_type := 'appendable_frame',
    values_cols := ['values_block_0', 'A', 'B', 'string', 'string2']]
/df/table (Table(30,)) ''
  description := {
  "index": Int64Col(shape=(), dflt=0, pos=0),
  "values_block_0": Float64Col(shape=(2,), dflt=0.0, pos=1),
  "A": Float64Col(shape=(), dflt=0.0, pos=2),
  "B": Float64Col(shape=(), dflt=0.0, pos=3),
  "string": StringCol(itemsize=3, shape=(), dflt='', pos=4),
  "string2": StringCol(itemsize=3, shape=(), dflt='', pos=5)}
  byteorder := 'little'
  chunkshape := (1424,)
  autoIndex := True
  colindexes := {
    "A": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "index": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "B": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "string2": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "string": Index(6, medium, shuffle, zlib(1)).is_CSI=False}
  /df/table._v_attrs (AttributeSet), 27 attributes:
   [A_dtype := 'float64',
    A_kind := ['A'],
    B_dtype := 'float64',
    B_kind := ['B'],
    CLASS := 'TABLE',
    FIELD_0_FILL := 0,
    FIELD_0_NAME := 'index',
    FIELD_1_FILL := 0.0,
    FIELD_1_NAME := 'values_block_0',
    FIELD_2_FILL := 0.0,
    FIELD_2_NAME := 'A',
    FIELD_3_FILL := 0.0,
    FIELD_3_NAME := 'B',
    FIELD_4_FILL := '',
    FIELD_4_NAME := 'string',
    FIELD_5_FILL := '',
    FIELD_5_NAME := 'string2',
    NROWS := 30,
    TITLE := '',
    VERSION := '2.6',
    index_kind := 'datetime64',
    string2_dtype := 'string24',
    string2_kind := ['string2'],
    string_dtype := 'string24',
    string_kind := ['string'],
    values_block_0_dtype := 'float64',
    values_block_0_kind := ['C', 'D']]
/df_dc (Group) ''
  /df_dc._v_attrs (AttributeSet), 12 attributes:
   [CLASS := 'GROUP',
    TITLE := '',
    VERSION := '1.0',
    data_columns := ['B', 'C', 'string', 'string2', 'datetime'],
    index_cols := [(0, 'index')],
    levels := 1,
    nan_rep := 'nan',
    non_index_axes := [(1, ['A', 'B', 'C', 'D', 'string', 'string2', 'datetime'])],
    pandas_type := 'frame_table',
    pandas_version := '0.10.1',
    table_type := 'appendable_frame',
    values_cols := ['values_block_0', 'B', 'C', 'string', 'string2', 'datetime']]
/df_dc/table (Table(30,)) ''
  description := {
  "index": Int64Col(shape=(), dflt=0, pos=0),
  "values_block_0": Float64Col(shape=(2,), dflt=0.0, pos=1),
  "B": Float64Col(shape=(), dflt=0.0, pos=2),
  "C": Float64Col(shape=(), dflt=0.0, pos=3),
  "string": StringCol(itemsize=3, shape=(), dflt='', pos=4),
  "string2": StringCol(itemsize=4, shape=(), dflt='', pos=5),
  "datetime": Int64Col(shape=(), dflt=0, pos=6)}
  byteorder := 'little'
  chunkshape := (1191,)
  autoIndex := True
  colindexes := {
    "index": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "C": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "B": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "string": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "string2": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "datetime": Index(6, medium, shuffle, zlib(1)).is_CSI=False}
  /df_dc/table._v_attrs (AttributeSet), 31 attributes:
   [B_dtype := 'float64',
    B_kind := ['B'],
    CLASS := 'TABLE',
    C_dtype := 'float64',
    C_kind := ['C'],
    FIELD_0_FILL := 0,
    FIELD_0_NAME := 'index',
    FIELD_1_FILL := 0.0,
    FIELD_1_NAME := 'values_block_0',
    FIELD_2_FILL := 0.0,
    FIELD_2_NAME := 'B',
    FIELD_3_FILL := 0.0,
    FIELD_3_NAME := 'C',
    FIELD_4_FILL := '',
    FIELD_4_NAME := 'string',
    FIELD_5_FILL := '',
    FIELD_5_NAME := 'string2',
    FIELD_6_FILL := 0,
    FIELD_6_NAME := 'datetime',
    NROWS := 30,
    TITLE := '',
    VERSION := '2.6',
    datetime_dtype := 'datetime64',
    datetime_kind := ['datetime'],
    index_kind := 'datetime64',
    string2_dtype := 'string32',
    string2_kind := ['string2'],
    string_dtype := 'string24',
    string_kind := ['string'],
    values_block_0_dtype := 'float64',
    values_block_0_kind := ['A', 'D']]

[jreback]

looks like it fully completed that test (the df_dc node is created at the end)....so weird that it would think there were 2 handles open....

[wesm]

I inserted return statements into the test and it starts failing after the final self.store.append at the end of the function-- if you end the test sooner it still works. So something bad going on inside there

[wesm]

to be perfectly clear:

        self.store.remove('df_dc')
        self.store.append('df_dc', df_dc, data_columns=['B', 'C',
                          'string', 'string2', 'datetime'])
        result = self.store.select('df_dc', [Term('B>0')])

If I put return after the first remove statement, everything is OK. Somehow another file handle is being created inside that append. weird

[wesm]

Or there is a reference to the handle that is not being garbage collected

[jreback]

weird...i have no easy way to test this....runs fine on linux (which generally is better about file handles anyhow)

[wesm]

Must be a bug of some kind in pytables. i don't have the time for a bisection exercise to find out where the dangling reference is created. skipping failure to os.remove file and the test passes now

[jreback]

k....i have a an amd64 build environment...trying to see if i find anything now....will let u know