Description
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
Code Sample, a copy-pastable example
import os
import pandas as pd
import numpy as np
file_contents = ("914793802660000000001000AAAAAAAAA0 "
+ "000000-00-000000-00-0000000000-00-0000000 "
+ " 0000000000-00-000000-00-00 000000000000000000.0000 0000000000000.00 00000\n")
header = [
'A',
'B',
'C',
'D',
'E',
'F',
'G',
'H',
'I',
'J',
'K',
'L',
'M',
'N',
'O',
'P',
'Q',
'R',
'S',
'T',
'U',
'V'
]
colspec_good = [
(0, 13),
(13, 23),
(23, 34),
(34, 45),
(45, 47),
(47, 57),
(57, 67),
(67, 71),
(71, 81),
(81, 86),
(86, 97),
(97, 103),
(103, 113),
(113, 123),
(123, 147),
(147, 164),
(164, 166),
(166, 168),
(168, 170),
(170, 172),
(172, 174),
(174, None)
]
print(len(header), len(colspec_good))
with open("bug.txt", "w") as foo:
foo.write(file_contents)
df = pd.read_fwf("bug.txt", names=header, colspecs=colspec_good, dtype=str)
print(df.head())
colspec_bad = [
(0, 13),
(13, 23),
(23, 34),
(34, 45),
(45, 47),
(47, 57),
(57, 67),
(67, 71),
(71, 81),
(81, 86),
(86, 97),
(97, 103),
(103, 113),
(113, 123),
(123, 147),
(147, 164),
(164, 166),
(166, 168),
(168, 170),
(170, 172),
(172, 174),
(174, 189),
(189, None)
]
print(len(header), len(colspec_bad))
with open("bug.txt", "w") as foo:
foo.write(file_contents)
df = pd.read_fwf("bug.txt", names=header, colspecs=colspec_bad, dtype=str)
with open("bug.txt", "w") as foo:
foo.write(file_contents * 2)
df = pd.read_fwf("bug.txt", names=header, colspecs=colspec_bad, dtype=str)
print(df.head())Problem description
When passing both the "names" and "colspecs" parameters, pandas.read_fwf fails to raise a warning/error when the length of the lists passed to these parameters does not match. In the given example, colspec_good has one tuple for every header name; colspec_bad has an extra 23rd tuple with no matching name. When the given file contains one row, a ParserError is raised with colspec_bad; but if given more than one row, no such warning/error is raised.
I propose that pandas.read_fwf should raise a warning/error when the length of the list passed to "names" doesn't match the length of the list passed to "colspecs" (when both parameters are used). Practically speaking, this tells the developer that they need to fix their header/colspecs. From a healthcare industry perspective, this has many nontrivial applications - e.g., the ingestion of standard, fixed-width CMS Medicare Advantage and CCLF (claim and claim line feed) files.
Expected Output
ParserError: Expected 22 fields in line 1, saw 23
Output of pd.show_versions()
INSTALLED VERSIONS
commit : f2c8480
python : 3.7.10.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None
pandas : 1.2.3
numpy : 1.19.2
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 52.0.0.post20210125
Cython : 0.29.22
pytest : 6.2.3
hypothesis : None
sphinx : 3.5.3
blosc : None
feather : None
xlsxwriter : 1.3.8
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.22.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 0.8.7
fastparquet : None
gcsfs : None
matplotlib : 3.3.4
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.2
sqlalchemy : 1.4.5
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.53.1