[BUG] Cannot find the `cmake` module in a wheel build venv when the `cmake` executable is installed in another venv

[XuehaiPan]

Expected Behavior

cmake should work with an arbitrary runtime environment when calling the cmake executable with an absolution path.

Current Behavior

When a venv (A) is installed with the cmake PyPI package, a venv/bin/cmake executable will present in PATH if the venv (A) is activated. While install a PyPI package from source, a build venv (B) will be created. In the build venv (B), run cmake installed in the venv (A) with absolute path will raise an error that cannot find the cmake module (installed in venv (A)).

After adding some debug script to the cmake executable:

  #!${PROJECT}/venvA/bin/python3.13
  # -*- coding: utf-8 -*-
  import re
  import sys
+ import site
+ from pprint import pformat
+
+ with open('debug.txt', 'w') as f:
+     print(f'site-packages: {pformat(site.getsitepackages())}', file=f)
+     print(f'sys.path: {pformat(sys.path)}', file=f)
+
  from cmake import cmake
  if __name__ == '__main__':
      sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
      sys.exit(cmake())

I got:

site-packages: ['${PROJECT}/venvA/lib/python3.13/site-packages']
sys.path: ['${PROJECT}/venvA/bin',
 '${TMP}/pip-build-env-xxx/site',
 '/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python313.zip',
 '/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13',
 '/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/lib-dynload',
 '${TMP}/pip-build-env-xxx/overlay/lib/python3.13/site-packages',
 '${TMP}/pip-build-env-xxx/normal/lib/python3.13/site-packages']

where the site-packages of venv (A) is not present in sys.path.

Reproduce Script

# setup.py

import os
import shutil
from pathlib import Path

from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext


HERE = Path(__file__).absolute().parent


class CMakeExtension(Extension):
    def __init__(self, name, source_dir=".", target=None, **kwargs):
        super().__init__(name, sources=[], **kwargs)
        self.source_dir = Path(source_dir).absolute()
        self.target = target if target is not None else name.rpartition(".")[-1]

    @classmethod
    def cmake_executable(cls):
        cmake = os.getenv("CMAKE_EXECUTABLE", "")
        if not cmake:
            cmake = shutil.which("cmake")
        return cmake


class cmake_build_ext(build_ext):
    def build_extension(self, ext):
        if not isinstance(ext, CMakeExtension):
            super().build_extension(ext)
            return

        cmake = ext.cmake_executable()
        if cmake is None:
            raise RuntimeError("Cannot find CMake executable.")

        self.spawn([cmake, "--version"])


setup(
    name="cmake-venv-test",
    version="0.0.1",
    cmdclass={"build_ext": cmake_build_ext},
    ext_modules=[CMakeExtension("cmake_venv_test._C", source_dir=HERE)],
)

$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install cmake
$ which cmake
${PROJECT}/venv/bin/cmake

$ pip3 install .
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing ${PROJECT}
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: cmake-venv-test
  Building wheel for cmake-venv-test (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for cmake-venv-test (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      running bdist_wheel
      running build
      running build_ext
      ${PROJECT}/venv/bin/cmake --version
      Traceback (most recent call last):
        File "${PROJECT}/venv/bin/cmake", line 5, in <module>
          from cmake import cmake
      ModuleNotFoundError: No module named 'cmake'
      error: command '${PROJECT}/venv/bin/cmake' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cmake-venv-test
Failed to build cmake-venv-test

ERROR: Failed to build installable wheels for some pyproject.toml based projects (cmake-venv-test)

Additional Context

Related:

• [BUG] optree tries to install cmake from PyPI unconditionally metaopt/optree#187
• feat(setup): support using system CMake metaopt/optree#188

The above issue and PR changed to use the cmake executable in PATH instead of adding the cmake PyPI package to build-system.requires in pyproject.toml.

cc @mgorny

[henryiii]

You should conditionally use cmake if it is not in your path and never put it into your build-system.requires. Scikit-build-core does this automatically, and has a wrapper for setuptools too. You can see a guide on how to do it yourself at https://scikit-build.readthedocs.io/en/latest/usage.html#adding-cmake-as-building-requirement-only-if-not-installed-or-too-low-a-version (really probably should be in the docs here!). That way you can support non-wheel supported platforms too, like FreeBSD, ClearLinux, etc.

However, I believe this is a bug, though I think this might be a bug with pip.

[henryiii]

I think this might be the same root issue as in scikit-build/scikit-build-core#955, which also looks like an issue with pip messing up the environment when calling a subprocess only when doing isolated installs.

[XuehaiPan]

However, I believe this is a bug, though I think this might be a bug with pip.

Yes. It should be an upstream bug for how project.scripts is created. Let me dig into it. This bug might affect all PyPI packages that provide executables, such as, cmake, ninja, clang-format, etc.

[XuehaiPan]

I opened an issue on the pip upstream:

• [BUG] Isolated build env breaks console scripts installed in other venv (e.g., cmake, ninja) pypa/pip#13222

[henryiii]

This only affects scripts that use an entry point to wrap them. Some of those directly place the executable inside the scripts directory, which would work. CMake has helper files (the directory that contains CMake modules, like Find*.cmake), which is why we have to have the wrapper. I think we might be able to put ninja directly in the scripts dir, but cmake is stuck like this AFAIK.

[henryiii]

Did you check using uv pip instead of pip to see if it has a similar problem? I'd expect not?

[XuehaiPan]

Did you check using uv pip instead of pip to see if it has a similar problem? I'd expect not?

FYI, uv pip works as expected.

[XuehaiPan]

A temporary workaround for pypa/pip#13222:

• [BUG] Isolated build env breaks console scripts installed in other venv (e.g., cmake, ninja) pypa/pip#13222

- self.spawn([cmake, '-S', str(ext.source_dir), '-B', str(build_temp), *cmake_args])
- if not self.dry_run:
-     self.spawn([cmake, '--build', str(build_temp), *build_args])
+ python_path = None
+ try:
+     # pip's build environment pseudo-isolation sets `PYTHONPATH` and may break console scripts
+     python_path = os.environ.pop('PYTHONPATH', None)  # unset `PYTHONPATH`
+     self.spawn([cmake, '-S', str(ext.source_dir), '-B', str(build_temp), *cmake_args])
+     if not self.dry_run:
+         self.spawn([cmake, '--build', str(build_temp), *build_args])
+ finally:
+     if python_path is not None:
+         os.environ['PYTHONPATH'] = python_path

[henryiii]

Working around this might be a bit ugly, but I think it's possible. We'd have to replace the entry point with a custom script that we install, then in the custom script we could use the same workaround that is present in https://github.com/astral-sh/ruff/pull/13591/files. (I'm pretty sure this is breaking down inside the script pip generates, and not inside __main__.py, which we certainly could easily patch?)

[XuehaiPan]

I'm pretty sure this is breaking down inside the script pip generates, and not inside __main__.py, which we certainly could easily patch?

I think we will need to update the distutils' console template. The error arises during importing the cmake module rather than finding the binary executable.

[henryiii]

Do you know what we'd need to add to the template? I have a replacement for it set up, but not sure how to safely check for and fix this case.

#!python
import os
import sys

# Scripts can be broken by pip's build isolation hacks
# See: https://github.com/pypa/pip/blob/102d8187a1f5a4cd5de7a549fd8a9af34e89a54f/src/pip/_internal/build_env.py#L87
# ?


from cmake import ${program}

if __name__ == '__main__':
    if sys.argv[0].endswith('-script.pyw'):
        sys.argv[0] = sys.argv[0][: -11]
    elif sys.argv[0].endswith('.exe'):
        sys.argv[0] = sys.argv[0][: -4]

    sys.exit(${program}())

[XuehaiPan]

Do you know what we'd need to add to the template?

I think the replacement should be:

#!python
import os
import sys

try:
    from cmake import ${program}
except ImportError:
    python_path = os.environ.pop('PYTHONPATH', None)
    from cmake import ${program}
    if python_path is not None:
        os.environ['PYTHONPATH'] = python_path

if __name__ == '__main__':
    if sys.argv[0].endswith('-script.pyw'):
        sys.argv[0] = sys.argv[0][: -11]
    elif sys.argv[0].endswith('.exe'):
        sys.argv[0] = sys.argv[0][: -4]

    sys.exit(${program}())

but not sure how to safely check for and fix this case.

Note that we should put PYTHONPATH back before invoking the true cmake binary. Because users may run Python commands in CMakeLists.txt. E.g.:

execute_process(
    COMMAND "${Python_EXECUTABLE}" -m pybind11 --cmakedir
    OUTPUT_VARIABLE pybind11_DIR
)