IndexError raised on invalid csv file

[Dominik1123]

Code Sample, a copy-pastable example if possible

$ python
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> pandas.__version__
'0.19.2'
>>> with open('/tmp/error.csv', 'w') as fp:
...     fp.write('+++123456789...\ncol1,col2,col3\n1,2,3\n')
... 
36
>>> pandas.DataFrame.from_csv('/tmp/error.csv')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dominik/.local/lib/python3.5/site-packages/pandas/core/frame.py", line 1231, in from_csv
    infer_datetime_format=infer_datetime_format)
  File "/home/dominik/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 646, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/dominik/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 401, in _read
    data = parser.read()
  File "/home/dominik/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 939, in read
    ret = self._engine.read(nrows)
  File "/home/dominik/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 1550, in read
    values = data.pop(self.index_col[i])
IndexError: list index out of range

Problem description

When loading a file with the following content via pandas.DataFrame.from_csv an IndexError is raised from somewhere inside the parsers sub-package.

File content:

+++123456789...
col1,col2,col3
1,2,3

Expected Output

The expected behavior would be to raise a CParserError instead.

Output of pd.show_versions()

>>> pandas.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-78-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 34.2.0
Cython: None
numpy: 1.12.0
scipy: 0.18.1
statsmodels: None
xarray: None
IPython: 5.2.2
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 2.0.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: 4.4.1
html5lib: 0.9999999
httplib2: 0.9.1
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.5
boto: None
pandas_datareader: None

[jreback]

@Dominik1123

In [1]: data = '+++123456789...\ncol1,col2,col3\n1,2,3\n'

In [2]: pd.read_csv(StringIO(data))
Out[2]: 
         +++123456789...
col1 col2            col3
1    2                  3

In [3]: pd.read_csv(StringIO(data), index_col=0)
IndexError: list index out of range

In [4]: pd.DataFrame.from_csv(StringIO(data))
IndexError: list index out of range

you probably want this

In [9]: pd.read_csv(StringIO(data), skiprows=1)
Out[9]: 
   col1  col2  col3
0     1     2     3

.from_csv defaults to index_col=0, e.g. [4] (which pd.read_csv does not, see [2]).

So this is buggy, and doing something odd.

cc @gfyoung

[Dominik1123]

@jreback Well actually the first line is just example garbage. It was an example for an invalid CSV file which a user might open in my application (so skipping the first row is no solution in my case). I'm excepting CParserError when loading the data frame (no matter if I load it with from_csv or read_csv(..., index_col=0). If other exceptions are raised (and this is not reported in the docs) it's hard to deal with such special cases. I expect to be able to except all parser errors with a single exception type.

[jreback]

@Dominik1123 hence the bug label

[gfyoung]

Can confirm this error on master (0f55de) with both engines.

Here's what is going on. When you pass in this data, there are clearly not enough columns (1) to hold 3 columns worth of data. As a result, pandas will move all extraneous column data to the DataFrame. It is for this reason that you get [2] from @jreback examples (that index is a MultiIndex). This behavior is to be expected.

If you specify index_col=0, you are saying that only the first column is the index, but then pandas doesn't know what to do with the middle column. It thinks that it should be used as part of the index, but then it is "surprised" when it finds out that you only wanted the first column to be the index, causing it to error.

Note that if you pass in index_col=[0, 1], DataFrame.from_csv works just fine. I think we can try to catch IndexError at that point and raise ParserError to be consistent. However, I'd call that an API issue, not a bug.

FYI @Dominik1123 , we have moved towards a more unified errors across both the Python and C engines and now raise ParserError instead of CParserError. They are currently aliases, but I would suggest you catch ParserError instead. Though to be fair, I'm not 100% sure where you saw that CParserError is raised in the docs? Also, read_csv is preferred over from_csv.

[roberthdevries]

@jbrockmendel Is this still an issue now that from_csv is deprecated?

[gfyoung]

@roberthdevries : If the issue does not replicate itself with read_csv, then I think we can close this one out with a test.

[roberthdevries]

I think the issue was never there with read_csv. As far as I understand this issue, there was a inconsistency between read_csv and from_csv where the former worked as expected. Now that one of the two parties is no longer around, there is also no inconsistency. Hence the ticket can be closed without further action.

[gfyoung]

Even so, we prefer to add a test for read_csv just in case. Can never have too many tests. 😉