read_csv c engine accepts binary mode data and python engine rejects it

[mgeens]

Code Sample

import pandas as pd

if __name__ == "__main__":
    with open('test.csv', 'w') as f:
        f.write('1,2,3\n4,5,6')
    with open('test.csv', 'rt') as f:
        pd.read_csv(f, header=None)
    with open('test.csv', 'rb') as f:
        pd.read_csv(f, header=None)
    with open('test.csv', 'rt') as f:
        pd.read_csv(f, header=None, engine='python')
    with open('test.csv', 'rb') as f:
        pd.read_csv(f, header=None, engine='python')

Problem description

The second read_csv call (using the C engine and a file opened in binary mode) will correctly read the csv. The fourth read_csv call (using the Python engine and a file opened in binary mode) will throw an exception stating it needs to be in text mode:

pandas.errors.ParserError: iterator should return strings, not bytes (did you open the file in text mode?)

Perhaps this is intended behavior, but I found this difference in behavior between the engines surprising, as well as that binary mode was accepted at all.

Expected Output

Either the C engine rejecting binary mode files or the Python engine accepting them.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.6.final.0 python-bits: 64 OS: Linux OS-release: 4.15.0-39-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 10.0.1
setuptools: 39.1.0
Cython: None
numpy: 1.15.4
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[mroeschke]

cc @gfyoung

[gfyoung]

@rgeens : Thanks for opening this! Sorry that this got lost in the pile of issues that we have 😞

I strongly believe that this discrepancy is symptomatic of limitations in Python's native csv library. The Python engine is largely a wrapper around it, and the error message you're showing comes from the csv module in fact.

To illustrate my point:

import csv

with open('test.csv', 'w') as f:
    f.write('1,2,3\n4,5,6')

r = csv.reader(open('test.csv', 'rb'))
print(next(r))

This outputs:

...
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

This is beyond our control, so the best we can do is to test the behavior for the C engine (if it doesn't exist already). You're more than welcome to do that!

[fiendish]

I would like to have this reopened, because it's not actually beyond pandas control to fix this cleanly and because right now the read_csv function doc specifies "file-like object" not "file-like object opened in ascii mode".

Just conditionally wrap the object in a io.TextIOWrapper, since you know that csv is by definition an ascii format:

import csv
import io

def __ascii_wrap(potentially_binary_buffer):
    try:
        return io.TextIOWrapper(potentially_binary_buffer)
    except Exception:
        return potentially_binary_buffer


with open('test.csv', 'w') as f:
    f.write('1,2,3\n4,5,6')

r = csv.reader(__ascii_wrap(open('test.csv', 'rb')))
print(next(r))