Description
When reading fixed-width files using the read_fwf
function, it is possible to specify a comment character using the comment
argument. I expected that all lines beginning with the comment character would be ignored. However, if you do not specify the first column in the file in any column in colspecs
, the comment character does not appear to be used.
Code Sample
import io
import pandas as pd
# Two input files, first line is comment, second line is data.
# Second file has a column (with the letter A)
# that I don't want at start of data.
string = "#\n1K\n"
off_string = "#\nA1K\n"
# When using skiprows to skip commented row, both work.
pd.read_fwf(io.StringIO(string), colspecs = [(0,1), (1,2)], skiprows = 1, header = None)
# 0 1
#0 1 K
pd.read_fwf(io.StringIO(off_string), colspecs = [(1,2), (2,3)], skiprows = 1, header = None)
# 0 1
#0 1 K
# If a comment character is specified, it only works when the colspecs
# includes the column with the comment character.
pd.read_fwf(io.StringIO(string), colspecs = [(0,1), (1,2)], comment = '#', header = None)
# 0 1
#0 1 K
pd.read_fwf(io.StringIO(off_string), colspecs = [(1,2), (2,3)], comment = '#', header = None)
# 0 1
#0 NaN NaN
#1 1.0 K
Expected Output
I expect that the final line should have the same result as the other three that is:
pd.read_fwf(io.StringIO(off_string), colspecs = [(1,2), (2,3)], comment = '#', header = None)
Should result in
# 0 1
#0 1 K
I first reported this on StackOverflow.
output of pd.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.6.5-x86_64-linode71
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
pandas: 0.18.1
nose: None
pip: 8.1.2
setuptools: 3.3
Cython: None
numpy: 1.11.1
scipy: None
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.14
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: None
boto: None
pandas_datareader: None