Closed
Description
Installation check
- I have read the installation guide.
Platform
Linux-6.10.4-linuxkit-armv7l-with
Installation Method
pip install
pandas Version
pandas-3.0.0.dev0+1524.g23c497bb2f
Python Version
3.12.5
Installation Logs
For Home Assistant we have an action to build custom musllinux wheels. Building pandas on armv7 started to fail recently with the following error:
[89/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o
FAILED: pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o
cc -Ipandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p -Ipandas/_libs -I../../pandas/_libs -I../../venv-312d/lib/python3.12/site-packages/numpy/core/include -I../../pandas/_libs/include -I/usr/local/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -DNPY_NO_DEPRECATED_API=0 -DNPY_TARGET_VERSION=NPY_1_21_API_VERSION -fPIC -MD -MQ pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o -MF pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o.d -o pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o -c ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c
In file included from /usr/include/fortify/string.h:26,
from ../../pandas/_libs/include/pandas/portable.h:12,
from ../../pandas/_libs/include/pandas/vendored/ujson/lib/ultrajson.h:55,
from ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:41:
/usr/include/fortify/stdlib.h:42:1: error: 'realpath' undeclared here (not in a function)
42 | _FORTIFY_FN(realpath) char *realpath(const char *__p, char *__r)
| ^~~~~~~~~~~
In file included from /usr/local/include/python3.12/Python.h:23,
from ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:43:
/usr/include/fortify/stdlib.h: In function 'realpath':
/usr/include/fortify/stdlib.h:45:2: error: #error PATH_MAX unset. A fortified realpath will not work.
45 | #error PATH_MAX unset. A fortified realpath will not work.
| ^~~~~
I bisected the issue to #55150 (cc @WillAyd), in particular the change in pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:
...
#define PY_ARRAY_UNIQUE_SYMBOL UJSON_NUMPY
#define NO_IMPORT_ARRAY
#define PY_SSIZE_T_CLEAN
+#include "pandas/vendored/ujson/lib/ultrajson.h"
#include <Python.h>
#include <numpy/arrayobject.h>
-#include "pandas/vendored/ujson/lib/ultrajson.h"
...It seems that, at least on the particular platform, the include order is important.
For the current main branch, it seems to be enough to modify this part:
pandas/pandas/_libs/src/vendored/ujson/python/JSONtoObj.c
Lines 41 to 43 in 23c497b
-#include "pandas/vendored/ujson/lib/ultrajson.h"
#define PY_SSIZE_T_CLEAN
+// clang-format off
#include <Python.h>
+#include "pandas/vendored/ujson/lib/ultrajson.h"
+// clang-format on/usr/src# pip install -v .
Using pip 24.2 from /usr/src/venv-312d/lib/python3.12/site-packages/pip (python 3.12)
Processing /usr/src
Running command Preparing metadata (pyproject.toml)
+ meson setup /usr/src /usr/src/.mesonpy-09kmtf13/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --vsenv --native-file=/usr/src/.mesonpy-09kmtf13/build/meson-python-native-file.ini
The Meson build system
Version: 1.2.1
Source dir: /usr/src
Build dir: /usr/src/.mesonpy-09kmtf13/build
Build type: native build
Project name: pandas
Project version: 3.0.0.dev0+1524.g23c497bb2f
C compiler for the host machine: cc (gcc 13.2.1 "cc (Alpine 13.2.1_git20240309) 13.2.1 20240309")
C linker for the host machine: cc ld.bfd 2.42
C++ compiler for the host machine: c++ (gcc 13.2.1 "c++ (Alpine 13.2.1_git20240309) 13.2.1 20240309")
C++ linker for the host machine: c++ ld.bfd 2.42
Cython compiler for the host machine: cython (cython 3.0.11)
Host machine cpu family: arm
Host machine cpu: armv7l
Program python found: YES (/usr/src/venv-312d/bin/python)
Found pkg-config: /usr/bin/pkg-config (2.2.0)
Run-time dependency python found: YES 3.12
Build targets in project: 54
pandas 3.0.0.dev0+1524.g23c497bb2f
User defined options
Native files: /usr/src/.mesonpy-09kmtf13/build/meson-python-native-file.ini
buildtype : release
vsenv : True
b_ndebug : if-release
b_vscrt : md
Found ninja-1.11.1.git.kitware.jobserver-1 at /usr/src/venv-312d/bin/ninja
Visual Studio environment is needed to run Ninja. It is recommended to use Meson wrapper:
/usr/src/venv-312d/bin/meson compile -C .
+ /usr/src/venv-312d/bin/ninja
[1/152] Generating pandas/_libs/index_class_helper_pxi with a custom command
[2/152] Generating pandas/_libs/khash_primitive_helper_pxi with a custom command
[3/152] Generating pandas/_libs/algos_common_helper_pxi with a custom command
[4/152] Generating pandas/_libs/hashtable_func_helper_pxi with a custom command
[5/152] Generating pandas/_libs/intervaltree_helper_pxi with a custom command
[6/152] Generating pandas/_libs/hashtable_class_helper_pxi with a custom command
[7/152] Generating pandas/_libs/algos_take_helper_pxi with a custom command
[8/152] Generating pandas/__init__.py with a custom command
[9/152] Generating pandas/_libs/sparse_op_helper_pxi with a custom command
[10/152] Generating write_version_file with a custom command
[11/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/base.pyx
[12/152] Compiling C object pandas/_libs/tslibs/parsing.cpython-312-arm-linux-musleabihf.so.p/.._src_parser_tokenizer.c.o
[13/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/ccalendar.pyx
[14/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/dtypes.pyx
[15/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/nattype.pyx
[16/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/np_datetime.pyx
[17/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/fields.pyx
[18/152] Compiling C object pandas/_libs/tslibs/base.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_base.pyx.c.o
[19/152] Linking target pandas/_libs/tslibs/base.cpython-312-arm-linux-musleabihf.so
[20/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/parsing.pyx
[21/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/conversion.pyx
[22/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/strptime.pyx
[23/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/timezones.pyx
[24/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/timedeltas.pyx
[25/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/offsets.pyx
[26/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/period.pyx
[27/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/timestamps.pyx
[28/152] Compiling C object pandas/_libs/tslibs/ccalendar.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_ccalendar.pyx.c.o
[29/152] Linking target pandas/_libs/tslibs/ccalendar.cpython-312-arm-linux-musleabihf.so
[30/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/tzconversion.pyx
[31/152] Compiling C object pandas/_libs/tslibs/np_datetime.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_np_datetime.pyx.c.o
[32/152] Linking target pandas/_libs/tslibs/np_datetime.cpython-312-arm-linux-musleabihf.so
[33/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/vectorized.pyx
[34/152] Compiling C object pandas/_libs/tslibs/dtypes.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_dtypes.pyx.c.o
[35/152] Linking target pandas/_libs/tslibs/dtypes.cpython-312-arm-linux-musleabihf.so
[36/152] Compiling C object pandas/_libs/tslibs/nattype.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_nattype.pyx.c.o
[37/152] Linking target pandas/_libs/tslibs/nattype.cpython-312-arm-linux-musleabihf.so
[38/152] Compiling Cython source /usr/src/pandas/_libs/arrays.pyx
[39/152] Compiling C object pandas/_libs/tslibs/conversion.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_conversion.pyx.c.o
[40/152] Linking target pandas/_libs/tslibs/conversion.cpython-312-arm-linux-musleabihf.so
[41/152] Compiling C object pandas/_libs/tslibs/timezones.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_timezones.pyx.c.o
[42/152] Linking target pandas/_libs/tslibs/timezones.cpython-312-arm-linux-musleabihf.so
[43/152] Compiling C object pandas/_libs/tslibs/fields.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_fields.pyx.c.o
[44/152] Linking target pandas/_libs/tslibs/fields.cpython-312-arm-linux-musleabihf.so
[45/152] Compiling C object pandas/_libs/tslibs/vectorized.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_vectorized.pyx.c.o
[46/152] Linking target pandas/_libs/tslibs/vectorized.cpython-312-arm-linux-musleabihf.so
[47/152] Compiling Cython source /usr/src/pandas/_libs/algos.pyx
[48/152] Compiling C object pandas/_libs/tslibs/tzconversion.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_tzconversion.pyx.c.o
[49/152] Compiling Cython source /usr/src/pandas/_libs/hashing.pyx
[50/152] Linking target pandas/_libs/tslibs/tzconversion.cpython-312-arm-linux-musleabihf.so
[51/152] Compiling C object pandas/_libs/arrays.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_arrays.pyx.c.o
[52/152] Linking target pandas/_libs/arrays.cpython-312-arm-linux-musleabihf.so
[53/152] Compiling C object pandas/_libs/tslibs/strptime.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_strptime.pyx.c.o
[54/152] Compiling Cython source /usr/src/pandas/_libs/indexing.pyx
[55/152] Linking target pandas/_libs/tslibs/strptime.cpython-312-arm-linux-musleabihf.so
[56/152] Compiling C object pandas/_libs/tslibs/parsing.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_parsing.pyx.c.o
[57/152] Linking target pandas/_libs/tslibs/parsing.cpython-312-arm-linux-musleabihf.so
[58/152] Compiling C object pandas/_libs/indexing.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_indexing.pyx.c.o
[59/152] Linking target pandas/_libs/indexing.cpython-312-arm-linux-musleabihf.so
[60/152] Compiling Cython source /usr/src/pandas/_libs/groupby.pyx
[61/152] Compiling Cython source /usr/src/pandas/_libs/internals.pyx
[62/152] Compiling Cython source /usr/src/pandas/_libs/index.pyx
[63/152] Compiling C object pandas/_libs/lib.cpython-312-arm-linux-musleabihf.so.p/src_parser_tokenizer.c.o
[64/152] Compiling C object pandas/_libs/tslibs/period.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_period.pyx.c.o
[65/152] Linking target pandas/_libs/tslibs/period.cpython-312-arm-linux-musleabihf.so
[66/152] Compiling Cython source /usr/src/pandas/_libs/hashtable.pyx
[67/152] Compiling Cython source /usr/src/pandas/_libs/missing.pyx
[68/152] Compiling C object pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so.p/src_vendored_numpy_datetime_np_datetime.c.o
[69/152] Compiling Cython source /usr/src/pandas/_libs/lib.pyx
[70/152] Compiling Cython source /usr/src/pandas/_libs/join.pyx
[71/152] Compiling C object pandas/_libs/hashing.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_hashing.pyx.c.o
[72/152] Linking target pandas/_libs/hashing.cpython-312-arm-linux-musleabihf.so
[73/152] Compiling C object pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so.p/src_vendored_numpy_datetime_np_datetime_strings.c.o
[74/152] Compiling C object pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so.p/src_datetime_date_conversions.c.o
[75/152] Compiling Cython source /usr/src/pandas/_libs/interval.pyx
[76/152] Compiling C object pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so.p/src_datetime_pd_datetime.c.o
[77/152] Linking target pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so
[78/152] Compiling C object pandas/_libs/pandas_parser.cpython-312-arm-linux-musleabihf.so.p/src_parser_io.c.o
[79/152] Compiling C object pandas/_libs/pandas_parser.cpython-312-arm-linux-musleabihf.so.p/src_parser_pd_parser.c.o
[80/152] Compiling C object pandas/_libs/pandas_parser.cpython-312-arm-linux-musleabihf.so.p/src_parser_tokenizer.c.o
[81/152] Linking target pandas/_libs/pandas_parser.cpython-312-arm-linux-musleabihf.so
[82/152] Compiling C object pandas/_libs/parsers.cpython-312-arm-linux-musleabihf.so.p/src_parser_tokenizer.c.o
[83/152] Compiling C object pandas/_libs/parsers.cpython-312-arm-linux-musleabihf.so.p/src_parser_io.c.o
[84/152] Compiling C object pandas/_libs/tslibs/timedeltas.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_timedeltas.pyx.c.o
[85/152] Linking target pandas/_libs/tslibs/timedeltas.cpython-312-arm-linux-musleabihf.so
[86/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_ujson.c.o
[87/152] Compiling C object pandas/_libs/tslibs/timestamps.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_timestamps.pyx.c.o
[88/152] Linking target pandas/_libs/tslibs/timestamps.cpython-312-arm-linux-musleabihf.so
[89/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o
FAILED: pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o
cc -Ipandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p -Ipandas/_libs -I../../pandas/_libs -I../../venv-312d/lib/python3.12/site-packages/numpy/core/include -I../../pandas/_libs/include -I/usr/local/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -DNPY_NO_DEPRECATED_API=0 -DNPY_TARGET_VERSION=NPY_1_21_API_VERSION -fPIC -MD -MQ pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o -MF pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o.d -o pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o -c ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c
In file included from /usr/include/fortify/string.h:26,
from ../../pandas/_libs/include/pandas/portable.h:12,
from ../../pandas/_libs/include/pandas/vendored/ujson/lib/ultrajson.h:55,
from ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:41:
/usr/include/fortify/stdlib.h:42:1: error: 'realpath' undeclared here (not in a function)
42 | _FORTIFY_FN(realpath) char *realpath(const char *__p, char *__r)
| ^~~~~~~~~~~
In file included from /usr/local/include/python3.12/Python.h:23,
from ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:43:
/usr/include/fortify/stdlib.h: In function 'realpath':
/usr/include/fortify/stdlib.h:45:2: error: #error PATH_MAX unset. A fortified realpath will not work.
45 | #error PATH_MAX unset. A fortified realpath will not work.
| ^~~~~
[90/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_lib_ultrajsonenc.c.o
[91/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_objToJSON.c.o
[92/152] Compiling Cython source /usr/src/pandas/_libs/parsers.pyx
[93/152] Compiling C object pandas/_libs/missing.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_missing.pyx.c.o
[94/152] Compiling C object pandas/_libs/internals.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_internals.pyx.c.o
[95/152] Compiling C object pandas/_libs/tslibs/offsets.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_offsets.pyx.c.o
[96/152] Compiling C object pandas/_libs/index.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_index.pyx.c.o
[97/152] Compiling C object pandas/_libs/lib.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_lib.pyx.c.o
[98/152] Compiling C object pandas/_libs/join.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_join.pyx.c.o
[99/152] Compiling C object pandas/_libs/interval.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_interval.pyx.c.o
[100/152] Compiling C object pandas/_libs/algos.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_algos.pyx.c.o
[101/152] Compiling C object pandas/_libs/groupby.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_groupby.pyx.c.o
[102/152] Compiling C object pandas/_libs/hashtable.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_hashtable.pyx.c.o
ninja: build stopped: subcommand failed.
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /usr/src/venv-312d/bin/python /usr/src/venv-312d/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp827zleqr
cwd: /usr/src
Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.