Skip to content

publish: Avoid redundant buffer copies #5903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 10, 2023

Conversation

Turbo87
Copy link
Member

@Turbo87 Turbo87 commented Jan 10, 2023

Previously, we were buffering the incoming hyper::Body into a bytes::Bytes instance, which we then wrapped in a Cursor, which we then used as a Read implementation to fill a Vec<u8>. This was somewhat wasteful because it meant that we were buffering the tarball payload twice: once in the Bytes and once more in the Vec<u8>.

This PR changes the implementation to take more advantage of the Arc-like behavior of bytes::Bytes. We implement a split_body() function, which turns one Bytes instance into two Bytes, one for the JSON metadata, and one for the tarball payload. This will just shuffle around a few pointers underneath, without performing any additional memory allocation.

In this PR we also slightly refactor the S3 uploader implementation to not require a redundant seek operation, which makes the above implementation a bit easier.

@Turbo87 Turbo87 added C-internal 🔧 Category: Nonessential work that would make the codebase more consistent or clear A-backend ⚙️ labels Jan 10, 2023
@Turbo87 Turbo87 merged commit b76f361 into rust-lang:master Jan 10, 2023
@Turbo87 Turbo87 deleted the publish-buffers branch January 10, 2023 17:43
@bors bors mentioned this pull request Jan 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-backend ⚙️ C-internal 🔧 Category: Nonessential work that would make the codebase more consistent or clear
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant