BUG: dataframe concatenation segementation fault (core dump)

[stokhos]

• [ X ] I have checked that this issue has not already been reported.

• [ X ] I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

    df = pd.concat(
        (
            pd.read_csv(os.path.join(dirpath, filename))
            for dirpath, filename in file_generator
        ),
        ignore_index=True,
    )

Problem description

A subgroup of my files can cause the above function to break. And it happens 100% with those files with pandas1.2.0, with 1.1.5 no problem.

[this should explain why the current behaviour is a problem and why the expected output is a better solution]

Expected Output

Output of pd.show_versions()

[paste the output of pd.show_versions() here leaving a blank line after the details tag]

INSTALLED VERSIONS ------------------ commit : 3e89b4c python : 3.9.1.final.0 python-bits : 64 OS : Linux OS-release : 5.9.16-200.fc33.x86_64 Version : #1 SMP Mon Dec 21 14:08:22 UTC 2020 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.2.0
numpy : 1.19.5
pytz : 2020.5
dateutil : 2.8.1
pip : 20.2.2
setuptools : 49.1.3
Cython : 0.29.21
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.5.1
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: 0.9.0
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.3
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : 0.8.7
xarray : None
xlrd : None
xlwt : None
numba : None

[phofl]

Please provide a reproducible example https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[stokhos]

@phofl Is there a way for me to pin down the bug on my side? It only happens to to files in two folder. I can't do the minimal bug report, because it has more than 1000 files.

[phofl]

You could do a git bisect to narrow down when this was introduced

IAlternatively it should be sufficientto use only these files? Maybe you could create the DataFrames casing the segfaults in code?

[jorisvandenbossche]

@stokhos a good start would probably to check whether the segfault happens in the concat function or actually in the read_csv (my guess would be the second). If you rewrite your code a little bit in multiple lines, you should be able to see on which line it fails.

In case it's caused by read_csv, you could then try to find out which of the 1000 files is failing to read (eg read them in a for loop and print the file path before reading, to see which one is failing)

[MarcoGorelli]

Closing for now, but please do let us know if you have a reproducible example, as per:

Please provide a reproducible example https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[djherbis]

I was able to reliably reproduce a segfault with a single row CSV on read_csv on pandas 1.2.0, however upgrading to 1.2.3 fixed it.

[MarcoGorelli]

@djherbis could you share that csv please?

[djherbis]

It's unfortunately got some private data in it, I'll try to narrow it down to single row, single column, and scrub the data so that it repros without private info.