Crashed Worker tracking issue

[TomAugspurger]

I've been seeing a lot of CI builds failing recently, but I have no idea what's causing them. Can maintainers add them to the table here when they occur?

CI Job	Test	Count
MacPython linux py_3.7_64	TestTableOrientReader::test_read_json_table_orient[vals0-index]	1
Windows py37_np18	testst_unique.py::test_unique_bad_unicode[idx_or_series_w_bad_unicode0]	1

[fangchenli]

I've been seeing a lot of CI builds failing recently, but I have no idea what's causing them. Can maintainers add them to the table here when they occur?
CI Job Test Count
MacPython linux py_3.7_64 TestTableOrientReader::test_read_json_table_orient[vals0-index] 1

I've encountered this one while working on #35214. It happened when the python version is 3.7.0. in my case. Bumping version to 3.7.1 fixed it. This specific test caused a Segmentation fault, therefore, crushed the worker. I reproduced it on my local machine using this environment.

name: pandas-dev
channels:
  - conda-forge
dependencies:
  - python=3.7.0

  # tools
  - cython=0.29.16
  - pytest=5.0.1
  - pytest-xdist>=1.21
  - hypothesis>=3.58.0
  - pytest-azurepipelines
  - psutil

  # pandas dependencies
  - beautifulsoup4=4.6.0
  - bottleneck=1.2.1
  - jinja2=2.10
  - numba=0.46.0
  - numexpr=2.6.8
  - numpy=1.16.0
  - openpyxl=2.5.7
  - pytables=3.4.4
  - python-dateutil=2.7.3
  - pytz=2017.3
  - scipy=1.2
  - xlrd=1.1.0
  - xlsxwriter=1.0.2
  - xlwt=1.3.0
  - html5lib=1.0.1

(pandas-test) fangchenli@fangchenli-home-office:~/Workspace/pandas-VirosaLi$ pytest pandas/tests/io/json/test_json_table_schema.py::TestTableOrient
============================================================================================================================================= test session starts =============================================================================================================================================
platform linux -- Python 3.7.0, pytest-5.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/fangchenli/Workspace/pandas-VirosaLi, inifile: setup.cfg
plugins: forked-1.2.0, xdist-1.33.0, azurepipelines-0.8.0, hypothesis-5.19.1
collected 47 items                                                                                                                                                                                                                                                                                            

pandas/tests/io/json/test_json_table_schema.py ..........................................Fatal Python error: Segmentation fault

Current thread 0x00007f54ac64d740 (most recent call first):
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 166 in __getitem__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 646 in _parse
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 426 in _parse_sub
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 816 in _parse
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 426 in _parse_sub
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 930 in parse
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_compile.py", line 764 in compile
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/re.py", line 286 in _compile
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/re.py", line 234 in compile
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/dataclasses.py", line 196 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "/home/fangchenli/Workspace/pandas-VirosaLi/pandas/core/dtypes/inference.py", line 420 in is_dataclass
  File "/home/fangchenli/Workspace/pandas-VirosaLi/pandas/core/frame.py", line 500 in __init__
  File "/home/fangchenli/Workspace/pandas-VirosaLi/pandas/tests/io/json/test_json_table_schema.py", line 619 in test_timestamp_in_columns
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/python.py", line 165 in pytest_pyfunc_call
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/python.py", line 1451 in runtest
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 117 in pytest_runtest_call
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 192 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 220 in from_call
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 192 in call_runtest_hook
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 167 in call_and_report
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 87 in runtestprotocol
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 72 in pytest_runtest_protocol
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 278 in pytest_runtestloop
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 257 in _main
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 213 in wrap_session
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 250 in pytest_cmdline_main
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/config/__init__.py", line 74 in main
  File "/home/fangchenli/miniconda3/envs/pandas-test/bin/pytest", line 11 in <module>
Segmentation fault (core dumped)

[TomAugspurger]

Interesting, thanks. The job at https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=39324&view=logs&j=266f9266-03bd-52fe-4891-1cfba8ecd83f&t=1190d2de-f7ad-53d5-9425-18b19f76e994 is using 3.7.0.

[WillAyd]

What's strange about this test is that if you look at the backtrace of a core dump the segfault seems to occur somewhere in the standard lib before ever executing any of the code. My guess is that there is a bug in 3.7.0 specifically with module initialization but haven't found anything conclusive. Strange that it would only happen with one test though...

[fangchenli]

I encountered the second failure listed in the table while working on #35214. But in a different environment.

https://travis-ci.org/github/pandas-dev/pandas/jobs/712331711

The environment is for the coverage test for python 3.7.

name: pandas-dev
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.7.*

  # tools
  - cython>=0.29.16
  - pytest>=5.0.1,<6.0.0rc0
  - pytest-xdist>=1.21
  - hypothesis>=3.58.0
  - pytest-cov  # this is only needed in the coverage build

  # pandas dependencies
  - beautifulsoup4
  - botocore>=1.11
  - cython>=0.29.16
  - dask
  - fastparquet>=0.3.2
  - fsspec>=0.7.4
  - gcsfs>=0.6.0
  - geopandas
  - html5lib
  - matplotlib
  - moto
  - nomkl
  - numexpr
  - numpy=1.16.*
  - odfpy
  - openpyxl
  - pandas-gbq
  - psycopg2
  - pyarrow>=0.13.0
  - pymysql
  - pytables
  - python-snappy
  - pytz
  - s3fs>=0.4.0
  - scikit-learn
  - scipy
  - sqlalchemy
  - statsmodels
  - xarray
  - xlrd
  - xlsxwriter
  - xlwt
  - pip
  - pip:
    - brotlipy
    - coverage
    - pandas-datareader
    - python-dateutil
    - pyxlsb

[mroeschke]

Looks like we've dropped 3.7 in our CI and the other builds haven't segfaulted in a while so I think we can close this issue for now. We can open a new issue if this occurs with another build