Skip to content

Add script to automatically download wheels #50859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jan 26, 2023
Merged

Add script to automatically download wheels #50859

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f2b77eb
Add script to automatically download wheels
datapythonista Jan 19, 2023
f8d45a1
Adding documentation to the script
datapythonista Jan 19, 2023
ba0fcbd
Improve usage message
datapythonista Jan 19, 2023
5f75b35
Fixing path of twine command
datapythonista Jan 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/source/development/maintaining.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -465,7 +465,8 @@ which will be triggered when the tag is pushed.

7. Download all wheels from the Anaconda repository where MacPython uploads them:
https://anaconda.org/multibuild-wheels-staging/pandas/files?version=<version>
to the ``dist/`` directory in the local pandas copy.
to the ``dist/`` directory in the local pandas copy. You can use the script
``scripts/download_wheels.sh`` to download all wheels at once.

8. Upload wheels to PyPI:

Expand Down
28 changes: 28 additions & 0 deletions scripts/download_wheels.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/sh
#
# Download all wheels for a pandas version.
#
# This script is mostly useful during the release process, when wheels
# generated by the MacPython repo need to be downloaded locally to then
# be uploaded to the PyPI.
#
# There is no API to access the wheel files, so the script downloads the
# website, extracts the file urls from the html, and then downloads it
# one by one to the dist/ directory where they would be generated.

VERSION=$1
DIST_DIR="$(realpath $(dirname -- $0)/../dist)"

if [ -z $VERSION ]; then
printf "Usage:\n\t$0 <version>\n\nWhere <version> is for example 1.5.3"
exit 1
fi

curl "https://anaconda.org/multibuild-wheels-staging/pandas/files?version=${VERSION}" | \
grep "href=\"/multibuild-wheels-staging/pandas/${VERSION}" | \
sed -r 's/.*<a href="([^"]+\.whl)">.*/\1/g' | \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not now, but in the future, might want to expand this to downloading the sdist too when we move to cibuildwheel(hopefully for 2.0).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no sdist now in https://anaconda.org/multibuild-wheels-staging/pandas/files?version=1.5.3 . We can add it by editing the regex if the new implementation adds it. We can also edit the release instructions at that point to not build the sdist locally.

awk '{print "https://anaconda.org" $0 }' | \
xargs wget -P $DIST_DIR

printf "\nWheels downloaded to $DIST_DIR\nYou can upload them to PyPI using:\n\n"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a small QOL improvement would be to print which wheels were downloaded?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if I know how to do it without making the script significantly more complex. I can just do a ls dist/ but assuming there were wheels in that directory it can report wrong information.

It's not super nice, but the log of wget shows the file names as they are downloaded. There is some noise in between files, but my preference would be to just have that. The alternatives I see are to save the url list in a temporary file, and then print the file content with a regex to remove the parts of the url that are not the file. Not worth in my opinion.

printf "\ttwine upload ${DIST_DIR}/pandas-${VERSION}*.{whl,tar.gz} --skip-existing"