installed cmake "binary" fails to run when used used as a pip install dependency

[nega0]

The installed cmake can't seem to find its own module when used as a pip install build dependency. This prevents the dependent package from being installed. For example:

❯ pip3 uninstall cmake
WARNING: Skipping cmake as it is not installed.

❯ pip3 install --verbose --no-cache-dir iminuit
Using pip 22.3.1 from /opt/homebrew/lib/python3.11/site-packages/pip (python 3.11)
Collecting iminuit
  Downloading iminuit-2.17.0.tar.gz (431 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 431.7/431.7 kB 9.3 MB/s eta 0:00:00
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=42
    Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
  Collecting cmake
    Using cached cmake-3.25.0-py2.py3-none-macosx_10_10_universal2.macosx_10_10_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl (45.1 MB)
  Installing collected packages: cmake, setuptools
  Successfully installed cmake-3.25.0 setuptools-65.6.3
  Installing build dependencies ... done
  [... skip ...]
  Building wheels for collected packages: iminuit
  Running command Building wheel for iminuit (pyproject.toml)
  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-12-arm64-cpython-311
  creating build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/_repr_text.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/experimental.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/version.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/_deprecated.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/util.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/__init__.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/minimize.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/pdg_format.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/cost.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/typing.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/testing.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/minuit.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/_repr_html.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  running build_ext
  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:
  Traceback (most recent call last):
    File "/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-build-env-8mge4o3a/overlay/bin/cmake", line 5, in <module>
      from cmake import cmake
  ModuleNotFoundError: No module named 'cmake'
[... skip ...]

❯ cat -n /private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-build-env-8mge4o3a/overlay/bin/cmake
     1	#!/opt/homebrew/opt/python@3.11/bin/python3.11
     2	# -*- coding: utf-8 -*-
     3	import re
     4	import sys
     5	from cmake import cmake
     6	if __name__ == '__main__':
     7	    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
     8	    sys.exit(cmake())

[HDembinski]

I cannot confirm this on my computer. I just installed iminuit in a clean python3.11 virtual environment and it worked fine. I am using a mac as well. I can see that you are using --no-cache-dir, but it seems pick up a cached cmake anyway. The cached cmake package does not seem to fit your distribution. You have MacOSX 12, but the cached file is for version 10. Perhaps that is the issue? Try cleaning the cache.

[nega0]

You'll observe that an older cached cmake wheel is not the problem. I suggest you also try it outside of a virtural environment.

❯ pip3 cache purge
WARNING: No matching packages
Files removed: 0

❯ pip3 install --verbose --no-cache-dir iminuit
Using pip 22.3.1 from /opt/homebrew/lib/python3.11/site-packages/pip (python 3.11)
Collecting iminuit
  Downloading iminuit-2.17.0.tar.gz (431 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 431.7/431.7 kB 9.2 MB/s eta 0:00:00
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=42
    Downloading setuptools-65.6.3-py3-none-any.whl (1.2 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 12.8 MB/s eta 0:00:00
  Collecting cmake
    Downloading cmake-3.25.0-py2.py3-none-macosx_10_10_universal2.macosx_10_10_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl (45.1 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.1/45.1 MB 20.9 MB/s eta 0:00:00
  Installing collected packages: cmake, setuptools
  Successfully installed cmake-3.25.0 setuptools-65.6.3
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:
  running egg_info
  writing src/iminuit.egg-info/PKG-INFO
  writing dependency_links to src/iminuit.egg-info/dependency_links.txt
  writing requirements to src/iminuit.egg-info/requires.txt
  writing top-level names to src/iminuit.egg-info/top_level.txt
  reading manifest file 'src/iminuit.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'CITATION'
  adding license file 'LICENSE'
  writing manifest file 'src/iminuit.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:
  running dist_info
  creating /private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-modern-metadata-ugxlvc69/iminuit.egg-info
  writing /private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-modern-metadata-ugxlvc69/iminuit.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-modern-metadata-ugxlvc69/iminuit.egg-info/dependency_links.txt
  writing requirements to /private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-modern-metadata-ugxlvc69/iminuit.egg-info/requires.txt
  writing top-level names to /private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-modern-metadata-ugxlvc69/iminuit.egg-info/top_level.txt
  writing manifest file '/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-modern-metadata-ugxlvc69/iminuit.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-modern-metadata-ugxlvc69/iminuit.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'CITATION'
  adding license file 'LICENSE'
  writing manifest file '/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-modern-metadata-ugxlvc69/iminuit.egg-info/SOURCES.txt'
  creating '/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-modern-metadata-ugxlvc69/iminuit-2.17.0.dist-info'
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy in /opt/homebrew/lib/python3.11/site-packages (from iminuit) (1.23.5)
Building wheels for collected packages: iminuit
  Running command Building wheel for iminuit (pyproject.toml)
  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-12-arm64-cpython-311
  creating build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/_repr_text.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/experimental.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/version.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/_deprecated.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/util.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/__init__.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/minimize.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/pdg_format.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/cost.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/typing.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/testing.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/minuit.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  copying src/iminuit/_repr_html.py -> build/lib.macosx-12-arm64-cpython-311/iminuit
  running build_ext
  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:
  Traceback (most recent call last):
    File "/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-build-env-9ey3ba7l/overlay/bin/cmake", line 5, in <module>
      from cmake import cmake
  ModuleNotFoundError: No module named 'cmake'
  cmake args: -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-install-oyjyfkpl/iminuit_c78717347f224ed8a679b9b779d614be/build/lib.macosx-12-arm64-cpython-311/iminuit/ -DPYTHON_EXECUTABLE=/opt/homebrew/Cellar/python@3.11/3.11.0/bin/python3.11 -DCMAKE_BUILD_TYPE=Release
  build args: --config Release -j8
  Traceback (most recent call last):
    File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
      main()
    File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/build_meta.py", line 413, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/build_meta.py", line 398, in _build_with_temp_dir
      self.run_setup()
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
      exec(code, locals())
    File "<string>", line 10, in <module>
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/dist.py", line 1208, in run_command
      super().run_command(command)
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/opt/homebrew/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 325, in run
      self.run_command("build")
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/dist.py", line 1208, in run_command
      super().run_command(command)
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 132, in run
      self.run_command(cmd_name)
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/dist.py", line 1208, in run_command
      super().run_command(command)
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
      _build_ext.run(self)
    File "/opt/homebrew/lib/python3.11/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
      _build_ext.build_ext.run(self)
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
      self.build_extensions()
    File "/opt/homebrew/lib/python3.11/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
      self._build_extensions_serial()
    File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
      self.build_extension(ext)
    File "/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-install-oyjyfkpl/iminuit_c78717347f224ed8a679b9b779d614be/cmake_ext.py", line 79, in build_extension
      subprocess.check_call(
    File "/opt/homebrew/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 413, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-install-oyjyfkpl/iminuit_c78717347f224ed8a679b9b779d614be', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-install-oyjyfkpl/iminuit_c78717347f224ed8a679b9b779d614be/build/lib.macosx-12-arm64-cpython-311/iminuit/', '-DPYTHON_EXECUTABLE=/opt/homebrew/Cellar/python@3.11/3.11.0/bin/python3.11', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
  error: subprocess-exited-with-error

  × Building wheel for iminuit (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /opt/homebrew/opt/python@3.11/bin/python3.11 /opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/tmp_6k8ib0t
  cwd: /private/var/folders/8p/q_dkyhqs3_dfyfh7hm9sql6h0000gn/T/pip-install-oyjyfkpl/iminuit_c78717347f224ed8a679b9b779d614be
  Building wheel for iminuit (pyproject.toml) ... error
  ERROR: Failed building wheel for iminuit
Failed to build iminuit
ERROR: Could not build wheels for iminuit, which is required to install pyproject.toml-based projects

[henryiii]

Pip on homebrew for CPython 3.10 and 3.11 is busted for isolated source builds. Fixed in Pip, but not yet released. This is true for anything you require when building - flit, hatchling, poetry, meson-python, scikit-build, cmake, ninja, all of it. It just happens that virtual environments have setuptools pre-installed when they are created (historical oddity that will eventually be removed), so that one dependency is okay, making it look like it's the fault of the others. Current workaround: use a venv, python@3.9, or avoid source builds.

[nega0]

Excellent. We already encourage our users to use python virtual environments, so the solution for us is stronger wording to that effect in our documentation.

For searchability's sake, it may be beneficial to keep this issue open until the pip fix appears in a release.

[nachovizzo]

I had the exact same issue, and while using pipx sometimes fix the issue, the workaround for me was to hack the skbuild on the setup.py:

import contextlib

@contextlib.contextmanager
def hack_skbuild():
    import shutil
    import os

    def cmd_exists(cmd):
        return shutil.which(cmd) is not None

    if not cmd_exists("cmake"):
        import cmake

        os.environ["PATH"] += os.pathsep + cmake.CMAKE_BIN_DIR

    if not cmd_exists("ninja"):
        import ninja

        os.environ["PATH"] += os.pathsep + ninja.BIN_DIR

    yield


with hack_skbuild():
    setup( #...)

In my case, as explained here I'm trying to build a package by only requiring pip and a c++ compiler to be available on the system.

The PEP 517 compliant build from scikit build actually works most of the time in my case, but it failed to "find" the cmake and ninja executables when any of them is not available on the $PATH. An early workaround was just to install them, but I wanted to also take in advance of the fact that they were already on the isolated environment created by pip

@henryiii do you think it makes sense I create a merge request by extending a bit the logic for this corner cases?

[kpedro88]

I am still encountering what appears to be the same error, even when using a venv.

Running the following commands in Python 3.10.16 (on AlmaLinux 8):

python3 -m venv venv
source venv/bin/activate
pip install --no-cache-dir --upgrade pip
pip install --no-cache-dir cmake
pip install --no-cache-dir xrootd

The last command (which requires cmake) produces the following log: log_pip_install_xrootd_cmake.log