do not upload tarballs, stop uploading wheels to multibuild-wheels-staging #164
Conversation
| --no-progress --force -u multibuild-wheels-staging \ | ||
| dist/scipy_openblas*.whl | ||
|
|
||
| fi |
There was a problem hiding this comment.
This was leftover from the transition to scientific-python-nightly-wheels
There was a problem hiding this comment.
No longer needed because releases go straight to PyPI now? Or do release builds go to the nighly bucket (unlike for numpy et al)?
There was a problem hiding this comment.
Wheels go to https://anaconda.org/scientific-python-nightly-wheels/ (below), this stanza was redundantly uploading to https://anaconda.org/multibuild-wheels-staging as well.
There was a problem hiding this comment.
Okay, if you're sure. For numpy/scipy/etc. the nightly wheels go to the first link, and releases to the latter. Since from this repo not every change turns into a new release, I had expected the same setup. Are you saying that for a release you download from the nightly bucket (by hand?) and then upload to PyPI?
It's possible I'm missing what happens for a release here, or that it's documented somewhere.
There was a problem hiding this comment.
I updated the build/release process in the README.md. The release wheels are uploaded manually from scientific-python-nighly-wheels. I think we should migrate numpy and scipy away from multibuild-wheels-staging as well, since the newer scientific-python-nightly-wheels container has a cleanup script to remove old files. We are getting close to the 50GB limit on multibuiild-wheels-staging, as can be seen by those with permissions here, duplicated below for those without permissions:
| Size | Artifact | Count |
|---|---|---|
| 15.4 GB | scipy | 448 files |
| 10.1 GB | openblas-libs | 561 files |
| 8.4 GB | pandas | 755 files |
| 4.5 GB | numpy | 330 files |
| 3.6 GB | statsmodels | 393 files |
| 1.3 GB | dipy | 178 files |
| 376.2 MB | scipy-openblas32 | 51 files |
| 318.7 MB | scipy-openblas64 | 43 files |
| 249.0 MB | yt | 24 files |
| 114.1 MB | biopython | 53 files |
| 97.5 MB | pywavelets | 24 files |
| 73.5 MB | astropy | 12 files |
| 44.0 MB | kiwisolver | 120 files |
Some of those files are old, do we really need the 1 year old scipy 1.9 wheels or the numpy 1.24.4 wheels? I removed any openblas-libs befor version 0.20, which saved about half its storage total.
There was a problem hiding this comment.
I updated the build/release process in the README.md.
thanks!
Some of those files are old, do we really need the 1 year old scipy 1.9 wheels or the numpy 1.24.4 wheels? I removed any openblas-libs befor version 0.20, which saved about half its storage total.
Problem solved by cleaning up scipy, took all of 3 minutes. Total storage now: 26.8 GB of 50.0 GB
That's over several years of usage, so doing nothing is fine for a long time again.
|
Copyinh from a review comment, I think we should also encourage projects using https://anaconda.org/multibuild-wheels-staging to move to https://anaconda.org/scietific-python-nightly-wheels. The latter has a script to remove old artifacts. |
I would caution that there are downsides as well. Not only is it more work to move over than clean up manually once every 2 years, but it's easier for the release manager to make a mistake when they have to manually copy ~30 wheels from a bucket where new nightly wheels may get uploaded daily or even in parallel with the release wheels. It's easy to miss one in that setup. I vote for doing nothing. |
|
As far as I can tell, both scipy and numpy use the same workflow as this repo: the wheels are downloaded via tools/download-wheels.py (for scipy) and tools/download-wheels.py (for numpy). So it should be simple to change those scripts when next someone cares enough about it. |
|
I will merge this to no longer upload the unused tarballs. |
|
Ah fair enough, forgot about |
xref scientific-python/upload-nightly-action#85
We don't have any more consumers of the tarballs, and the wheels have proved sufficient.
cc @rgommers