Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import tarfile
with open("test.csv", "wt") as file:
file.write("1;2\n3;4")
with tarfile.open("test-csv.tar", "w") as archive:
archive.add("test.csv")
import pandas as pd
data = pd.read_csv("tar://test.csv::file://test-csv.tar", compression=None) # works
print(data)
data = pd.read_csv("tar://test.csv::file://test-csv.tar") # does not workIssue Description
For chained URLs, the file gets misidentified as TAR, which leads to this backtrace:
Traceback (most recent call last):
File "/projects/ratarmount/worktrees/1/trigger-pandas-bug.py", line 12, in <module>
data = pd.read_csv("tar://test.csv::file://test-csv.tar", compression="infer") # does not work
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/.local/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
return _read(filepath_or_buffer, kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/.local/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 620, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/.local/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
self._engine = self._make_engine(f, self.engine)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/.local/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
self.handles = get_handle(
^^^^^^^^^^^
File "~/.local/lib/python3.12/site-packages/pandas/io/common.py", line 828, in get_handle
handle = _BytesTarFile(
^^^^^^^^^^^^^^
File "~/.local/lib/python3.12/site-packages/pandas/io/common.py", line 991, in __init__
self.buffer: tarfile.TarFile = tarfile.TarFile.open( # type: ignore[assignment]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/tarfile.py", line 1842, in open
raise ReadError(f"file could not be opened successfully:\n{error_msgs_summary}")
tarfile.ReadError: file could not be opened successfully:
- method gz: ReadError('not a gzip file')
- method bz2: ReadError('not a bzip2 file')
- method xz: ReadError('not an lzma file')
- method tar: ReadError('truncated header')I have checked the source code, and the problem seems to be that the full URL is checked for ending with a TAR extension. Instead, only the last part in the chain should be checked, i.e., it should check the extension of tar://test.csv not tar://test.csv::file://test-csv.tar.
Lines 593 to 594 in 2a10e04
Expected Behavior
It should work without an error.
Installed Versions
INSTALLED VERSIONS
commit : 2a10e04
python : 3.12.3
python-bits : 64
OS : Linux
OS-release : 6.8.0-45-generic
Version : #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 3.0.0.dev0+1579.g2a10e04a0
numpy : 1.26.4
dateutil : 2.9.0.post0
pip : 24.2
Cython : 3.0.11
sphinx : None
IPython : 8.20.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
fastparquet : None
fsspec : 2024.9.0
html5lib : 1.1
hypothesis : None
gcsfs : 2024.9.0post1
jinja2 : 3.1.4
lxml.etree : 5.2.1
matplotlib : 3.6.3
numba : None
numexpr : None
odfpy : None
openpyxl : None
psycopg2 : None
pymysql : None
pyarrow : 15.0.2
pyreadstat : None
pytest : 8.3.3
python-calamine : None
pytz : 2024.1
pyxlsb : None
s3fs : 2024.9.0
scipy : 1.11.4
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
xlsxwriter : None
zstandard : 0.23.0
tzdata : 2024.1
qtpy : None
pyqt5 : None