ZipFile.writestr should respect SOURCE_DATE_EPOCH

[SomberNight]

BPO	47123
Nosy	@SomberNight
Files	zipfile_respect_sourcedate.diff: change the ZipFile.writestr to respect SOURCE_DATE_EPOCH

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-25.19:47:10.113>
labels = ['type-feature', 'library', '3.9', '3.10', '3.11']
title = 'ZipFile.writestr should respect SOURCE_DATE_EPOCH'
updated_at = <Date 2022-03-25.19:47:10.113>
user = 'https://github.com/SomberNight'

bugs.python.org fields:

activity = <Date 2022-03-25.19:47:10.113>
actor = 'ghost43'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-03-25.19:47:10.113>
creator = 'ghost43'
dependencies = []
files = ['50702']
hgrepos = []
issue_num = 47123
keywords = ['patch']
message_count = 1.0
messages = ['416015']
nosy_count = 1.0
nosy_names = ['ghost43']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue47123'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-91279: ZipFile.writestr now respect SOURCE_DATE_EPOCH #124435

[SomberNight]

Currently ZipFile.writestr writes the local time into the ZipFile.
(depends on both current time and local timezone)
See

cpython/Lib/zipfile.py

Lines 1816 to 1817 in 20e6e56

	zinfo = ZipInfo(filename=zinfo_or_arcname,
	date_time=time.localtime(time.time())[:6])

This makes pip installing a package generate non-reproducible build artifacts.

Specifically, Scripts/*.exe files (created for packages that define entry_points/console_scripts) are not reproducible on Windows when installed by pip. This also leaks into the *.dist-info/RECORD files.

For example, after running pip install wheel or pip install pyinstaller,
in wheel-0.37.1.dist-info/RECORD, I have this line:

../../Scripts/wheel.exe,sha256=u9TbPw2XNs_F9uy7y2zwumuzAZDbOSB7BXjLHZ0tTHg,97103

in pyinstaller-4.10.dist-info/RECORD, I have these lines:

../../Scripts/pyi-archive_viewer.exe,sha256=nC-9muPlIhUC1qvFkXHpyKJyRQqXISXxbUPXQ1XVOiM,97133
../../Scripts/pyi-bindepend.exe,sha256=udFHiAdndPpSwaIqmhmLEy36IUs1cNNoNQznSEnLJQQ,97128
../../Scripts/pyi-grab_version.exe,sha256=3ET9E841tFWujFL99aG4frzgwlP9f9pAkMgE0k2UGK0,97131
../../Scripts/pyi-makespec.exe,sha256=dJkfmITdLJhyPngmqziqqj5tH9qqfeQc5BTubeoXWUs,97127
../../Scripts/pyi-set_version.exe,sha256=sWmcOVS93fUY-wbdoz6ixBCvjy1tC4Aaw30DMmrmo-0,97130
../../Scripts/pyinstaller.exe,sha256=haInbhH0pImJn24cW4v917oUZmzXZj8OE89KFh4MO2Y,97112

Upon comparing multiple Scripts/wheel.exe files, I've found that the only difference is due to the above-mentioned timestamp embedded inside the exe (or rather, same timestamp embedded twice).

The exe files get created by distlib (vendored by pip).
Here is a traceback with an artificial exception to illustrate the codepath:

(env) PS C:\tmp> pip install --no-build-isolation pyinstaller
Collecting pyinstaller
  Using cached pyinstaller-4.10-py3-none-win_amd64.whl (2.0 MB)
Requirement already satisfied: setuptools in c:\tmp\env\lib\site-packages (from pyinstaller) (61.0.0)
Requirement already satisfied: pyinstaller-hooks-contrib>=2020.6 in c:\tmp\env\lib\site-packages (from pyinstaller) (2022.3)
Requirement already satisfied: altgraph in c:\tmp\env\lib\site-packages (from pyinstaller) (0.17.2)
Requirement already satisfied: pefile>=2017.8.1 in c:\tmp\env\lib\site-packages (from pyinstaller) (2021.9.3)
Requirement already satisfied: pywin32-ctypes>=0.2.0 in c:\tmp\env\lib\site-packages (from pyinstaller) (0.2.0)
Requirement already satisfied: future in c:\tmp\env\lib\site-packages (from pefile>=2017.8.1->pyinstaller) (0.18.2)
Installing collected packages: pyinstaller
ERROR: Exception:
Traceback (most recent call last):
  File "C:\tmp\env\lib\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "C:\tmp\env\lib\site-packages\pip\_internal\cli\req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "C:\tmp\env\lib\site-packages\pip\_internal\commands\install.py", line 405, in run
    installed = install_given_reqs(
  File "C:\tmp\env\lib\site-packages\pip\_internal\req\__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "C:\tmp\env\lib\site-packages\pip\_internal\req\req_install.py", line 769, in install
    install_wheel(
  File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 729, in install_wheel
    _install_wheel(
  File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 646, in _install_wheel
    generated_console_scripts = maker.make_multiple(scripts_to_generate)
  File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 440, in make_multiple
    filenames.extend(self.make(specification, options))
  File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 427, in make
    return super().make(specification, options)
  File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 429, in make
    self._make_script(entry, filenames, options=options)
  File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 329, in _make_script
    self._write_script(scriptnames, shebang, script, filenames, ext)
  File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 263, in _write_script
    raise Exception(f"heyheyhey2. {sha256(launcher)=}. {sha256(shebang)=}. {sha256(zip_data)=}. " +
Exception: heyheyhey2. sha256(launcher)='a00a877acefc'. sha256(shebang)='58628e924f22'. sha256(zip_data)='a423496a0482'. ('SOURCE_DATE_EPOCH' in os.environ)=True

The interesting code is here:
https://github.com/pypa/distlib/blob/d0e3f49df5d1aeb9daeaaabf0391c9e13e4a6562/distlib/scripts.py#L251-L252
This calls into the cpython standard library, where time.time() gets written into the file:

cpython/Lib/zipfile.py

Lines 1816 to 1817 in 20e6e56

	zinfo = ZipInfo(filename=zinfo_or_arcname,
	date_time=time.localtime(time.time())[:6])

Ideally, either distlib or the stdlib zipfile module should be changed to respect SOURCE_DATE_EPOCH, but it's not entirely clear to me which...

The attached patch changes ZipFile.writestr to respect SOURCE_DATE_EPOCH if set.

related pypa/distlib#164
related spesmilo/electrum#7739