Skip to content

read_fwf does not use comment character if colspecs argument does not include first column #14135

Closed
@nograpes

Description

@nograpes

When reading fixed-width files using the read_fwf function, it is possible to specify a comment character using the comment argument. I expected that all lines beginning with the comment character would be ignored. However, if you do not specify the first column in the file in any column in colspecs, the comment character does not appear to be used.

Code Sample

import io
import pandas as pd

# Two input files, first line is comment, second line is data.
# Second file has a column (with the letter A) 
# that I don't want at start of data.
string = "#\n1K\n"
off_string = "#\nA1K\n"

# When using skiprows to skip commented row, both work.
pd.read_fwf(io.StringIO(string), colspecs = [(0,1), (1,2)], skiprows = 1, header = None)
#    0  1
#0  1  K

pd.read_fwf(io.StringIO(off_string), colspecs = [(1,2), (2,3)], skiprows = 1, header = None)
#    0  1
#0  1  K

# If a comment character is specified, it only works when the colspecs 
# includes the column with the comment character.
pd.read_fwf(io.StringIO(string), colspecs = [(0,1), (1,2)], comment = '#', header = None)
#    0  1
#0  1  K

pd.read_fwf(io.StringIO(off_string), colspecs = [(1,2), (2,3)], comment = '#', header = None)
#      0    1
#0  NaN  NaN
#1  1.0    K

Expected Output

I expect that the final line should have the same result as the other three that is:

pd.read_fwf(io.StringIO(off_string), colspecs = [(1,2), (2,3)], comment = '#', header = None)

Should result in

#    0  1
#0  1  K

I first reported this on StackOverflow.

output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.6.5-x86_64-linode71
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1
nose: None
pip: 8.1.2
setuptools: 3.3
Cython: None
numpy: 1.11.1
scipy: None
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.14
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: None
boto: None
pandas_datareader: None

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions