Skip to content

do not upload tarballs, stop uploading wheels to multibuild-wheels-staging #164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 18, 2024
Merged

do not upload tarballs, stop uploading wheels to multibuild-wheels-staging #164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4743a28
do not upload tarballs, stop uploading wheels to multibuild-wheels-st…
mattip Jul 16, 2024
8b26e6f
update README.md
mattip Jul 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 13 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,19 @@
We build OpenBLAS on Travis-CI (for linux aarch64, ppc64, s390x) and github actions
for linux, windows, macOS x86_64 and macOS arm64.

Tarballs are at
https://anaconda.org/scientific-python-nightly-wheels/openblas-libs/files

A project using the tarball, for Manylinux or macOS, will need the
``gfortran-install`` submodule used here, from
https://github.com/MacPython/gfortran-install

We also build and upload a pip-installable wheel. The wheel is self-contained,
it includes all needed gfortran support libraries. On windows, this is a single
DLL. On linux we use `auditwheel repair` to mangle the shared object names.
First, tarballs are built using `do_build_lib` in `tools/build_steps.sh` (on
posix in a docker and drectly on macos) or `build_openblas.sh` on windows.

Then the shared object and header files from the tarball are used to build the
wheel via `tools/build_wheel.sh`, and the wheels uploaded to
https://anaconda.org/scientific=python-nightly-wheels/scipy_openblas32 and
https://anaconda.org/scientific=python-nightly-wheels/scipy_openblas64 via
`tools/upload_to_anaconda_staging.sh`. For a release, the wheels are uploaded
to PyPI by downloading them via tools/dowlnload-wheels.py and uploading via
[twine](https://twine.readthedocs.io/en/stable/).

The wheel is self-contained, it includes all needed gfortran support libraries.
On windows, this is a single DLL.

The wheel supplies interfaces for building and using OpenBLAS in a python
project like SciPy or NumPy:
Expand Down
14 changes: 0 additions & 14 deletions tools/build_steps.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,17 +203,3 @@ function do_build_lib {
$BUILD_PREFIX/lib/pkgconfig/scipy-openblas* \
$BUILD_PREFIX/lib/cmake/openblas
}

function upload_to_anaconda {
if [ "$ANACONDA_SCIENTIFIC_PYTHON_UPLOAD" == "" ]; then
echo "ANACONDA_SCIENTIFIC_PYTHON_UPLOAD is not defined: skipping."
else
anaconda -t $ANACONDA_SCIENTIFIC_PYTHON_UPLOAD upload \
--no-progress --force -u scientific-python-nightly-wheels \
-t file -p "openblas-libs" \
-v "$(cd OpenBLAS && git describe --tags --abbrev=8)" \
-d "OpenBLAS for multibuild wheels" \
-s "OpenBLAS for multibuild wheels" \
libs/openblas*.tar.gz
fi
}
16 changes: 0 additions & 16 deletions tools/upload_to_anaconda_staging.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,6 @@ upload_wheels() {
VERSION=$(git describe --tags --abbrev=8)
popd

if [ "$OPENBLAS_LIBS_STAGING_UPLOAD_TOKEN" != "" ]; then
echo "Uploading OpenBLAS $VERSION to anaconda.org/multibuild-wheels-staging"

anaconda -t $OPENBLAS_LIBS_STAGING_UPLOAD_TOKEN upload \
--no-progress --force -u multibuild-wheels-staging \
dist/scipy_openblas*.whl

fi
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was leftover from the transition to scientific-python-nightly-wheels

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed because releases go straight to PyPI now? Or do release builds go to the nighly bucket (unlike for numpy et al)?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wheels go to https://anaconda.org/scientific-python-nightly-wheels/ (below), this stanza was redundantly uploading to https://anaconda.org/multibuild-wheels-staging as well.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, if you're sure. For numpy/scipy/etc. the nightly wheels go to the first link, and releases to the latter. Since from this repo not every change turns into a new release, I had expected the same setup. Are you saying that for a release you download from the nightly bucket (by hand?) and then upload to PyPI?

It's possible I'm missing what happens for a release here, or that it's documented somewhere.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the build/release process in the README.md. The release wheels are uploaded manually from scientific-python-nighly-wheels. I think we should migrate numpy and scipy away from multibuild-wheels-staging as well, since the newer scientific-python-nightly-wheels container has a cleanup script to remove old files. We are getting close to the 50GB limit on multibuiild-wheels-staging, as can be seen by those with permissions here, duplicated below for those without permissions:

Size Artifact Count
15.4 GB scipy 448 files
10.1 GB openblas-libs 561 files
8.4 GB pandas 755 files
4.5 GB numpy 330 files
3.6 GB statsmodels 393 files
1.3 GB dipy 178 files
376.2 MB scipy-openblas32 51 files
318.7 MB scipy-openblas64 43 files
249.0 MB yt 24 files
114.1 MB biopython 53 files
97.5 MB pywavelets 24 files
73.5 MB astropy 12 files
44.0 MB kiwisolver 120 files

Some of those files are old, do we really need the 1 year old scipy 1.9 wheels or the numpy 1.24.4 wheels? I removed any openblas-libs befor version 0.20, which saved about half its storage total.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the build/release process in the README.md.

thanks!

Some of those files are old, do we really need the 1 year old scipy 1.9 wheels or the numpy 1.24.4 wheels? I removed any openblas-libs befor version 0.20, which saved about half its storage total.

Problem solved by cleaning up scipy, took all of 3 minutes. Total storage now: 26.8 GB of 50.0 GB

That's over several years of usage, so doing nothing is fine for a long time again.

if [ "$ANACONDA_SCIENTIFIC_PYTHON_UPLOAD" == "" ]; then
echo "ANACONDA_SCIENTIFIC_PYTHON_UPLOAD is not defined: skipping."
else
Expand All @@ -29,13 +21,5 @@ upload_wheels() {
anaconda -t $ANACONDA_SCIENTIFIC_PYTHON_UPLOAD upload \
--no-progress --force -u scientific-python-nightly-wheels \
dist/scipy_openblas*.whl

tarballs=$(ls -d builds/openblas*.zip libs/openblas*.tar.gz 2>/dev/null)
anaconda -t $ANACONDA_SCIENTIFIC_PYTHON_UPLOAD upload \
--no-progress --force -u scientific-python-nightly-wheels \
-t file -p "openblas-libs" -v "$VERSION" \
-d "OpenBLAS for multibuild wheels" \
-s "OpenBLAS for multibuild wheels" \
${tarballs}
fi
}
Loading