Look into ways to decouple some crate metadata from `Cargo.toml`

[jtgeibel]

An outline of our current scheme for crate and version level metadata (with quoted fields coming from Cargo.toml:

• Crate-wide metadata
  • description
  • homepage
  • documentation
  • repository
  • categories
  • keywords
  • badges (largely deprecated)
  • readme (we cache the most recent readme for search)
• Version specific metadata
  • features
  • license
  • crate_size
  • published_by
  • authors

With the exception of authors, all of the version specific metadata is clearly immutably tied to the specific crate that was published under that name and version. For the crate-wide metadata, it can be annoying to be required to publish a new version to update these fields.

It would be nice if the crate-wide metadata was managed through the web interface (Edit: and eventually a stable API to support cargo integration and other tooling), instead of via Cargo.toml.

Suggestions

For optional fields, we can probably do something like:

• If the latest stable doesn't specify the field, then it can be added/edited in the frontend UI. If the field is defined, the frontend can direct the user to first publish a new version without the field.
• If a new stable release overwrites the field by adding it to Cargo.toml, then we overwrite the value and return the previous contents in a warning to the user. (If this was a mistake, then the user has to publish a new crate that is otherwise a no-op. This is a bit unfortunate, but seems better than forcing a new publish to change any piece of metadata.)

The only required crate-wide field is description. I don't see changing this as high-priority, so we can focus on the optional fields.

A few additional notes

We've had bugs in the past where publishing 0.2.1 and backporting to 0.1.12 would overwrite metadata unexpectedly if they were published in that order. Off the top of my head, I don't know exactly how we handle pre-releases and metadata updates. My point is, saving the metadata from the "right" release isn't entirely straightforward and might cause unexpected results. It would be nice if we could reduce the user impact of this complexity.

The documentation field is a bit interesting, because there is no way for a crate version to link directly to version specific information. We fixup any docs.rs links automatically in the frontend so that they automatically point to the correct version specific documentation (matching the behavior for when a value is not provided). This means that user can either have automatic version-specific links to docs.rs, or one link to their most recent 3rd party documentation for all their releases. (Edit: I think we could improve this field by supporting some type of {$version} placeholder where crates.io would automatically customize the URL for each release but it would only need to be set once globally for the crate and could be easily modified if a project's 3rd party documentation moves.)

[Diggsey]

I would still like the non-versioned information to be managed through cargo, not through a crates.io-specific interface. (Although it's fine if crates.io also has a way to manage that as well). This would allow that information to be easily updated by CI/CD pipelines, and could be managed in a way that is not tied specifically to crates.io.

This is why I suggested putting that information in a separate file, which is also read by cargo publish.

[Ekleog]

Random thought: the information could also be written in Cargo.toml, and just stripped out of it by cargo publish before uploading to the repo?

Not sure whether this idea holds its weight, though, but it'd avoid most if not all the issues around deprecating the authors field I saw discussed in the RFC there.

[SOF3]

Currently crates.io cleans Cargo.toml with Cargo.toml.orig in a separate file. What about changing them to Cargo-version.toml and Cargo-package.toml in two separate files?

[jtgeibel]

Currently crates.io cleans Cargo.toml with Cargo.toml.orig in a separate file.

This is done by cargo actually.

I would still like the non-versioned information to be managed through cargo, not through a crates.io-specific interface.

I would also like to provide a stable API and maybe a cargo package-meta command or something. I certainly see the advantage of being able to modify this metadata in a stable way that can be accessed via tooling, but I think building this into the website first would satisfy most initial use-cases. Over the years there has been some discussion of a /api/v2/ namespace where we could provide better a better long-term API and eventually fix some warts in our existing APIs. I'd love to see something similar to the alternative registries RFC to stabilize some APIs around managing package-level metadata.

[kornelski]

Please keep alternative registries in mind. If the solution is crates-io-specific, then other registries will struggle to manage this data.

This data could theoretically be added to the git-based registry index, but the amount of extra data could be painful. However, the planned HTTP-based registry protocol could accommodate extra data. rust-lang/rfcs#2789

[VulpineAmethyst]

Regarding the author field: my opinion is that it should be injected in to the crate when it is fetched. This would allow the field to change as needed, retroactively, without requiring updating and resigning every affected crate.

[wizzwizz4]

It might help to think of this abstractly. We've got some data that is:

• Constant (e.g. crate names)
• Historical (e.g. the code that was published)
• Mutable (e.g. the latest unyanked version)

I think this is roughly how we're thinking of it at the moment. However, this isn't expressive enough to effectively represent all the information we want. There are other properties to consider:

• Mutable? (Does it change?)
• Historical? (Will we keep the historical record?)
• Certain? (Are we free from needing to correct the record?)
• Revocable? (Might we need to remove access to it? Includes yanking.)
• Indirect? (Does it refer to the thing we actually want to keep track of?)

(This list might not be exhaustive.) I've had a go at placing some things into this framework:

	M	H	C	R	I
crate name			✔️
current version	✔️		✔️
a release		✔️	✔️	✔️
a dependency		✔️	✔️		✔️: CR
an version's author		✔️		✔️
an author's current name	✔️		✔️	✔️	✔️: HR
contact details	✔️			✔️
download statistics	✔️	✔️		✔️

I haven't described indirection well enough here, because it has three levels of potential mutability:

• A change in the reference (e.g. changing mycrate:3.0 to mycrate:3.1)
• A change in the referencee (I have no examples, so *x = 5 is my example)
• A change in how the reference is looked up (e.g. mycrate version 3.1.2 is released, so mycrate:3.1's meaning has changed)

Are there any glaring holes in this framework?

[wizzwizz4]

Hmm… Really, we only need to keep things that affect builds immutable:

• The public API;
• The code's behaviour (including interaction with undocumented aspects of rustc); and
• Anything the build script reads. (For most scripts, it should be possible to find this statically, but it's not solvable in general. Build scripts already cause problems for immutable builds, though, so this probably isn't too big of a concern.)

We'd probably need guarantees from rustc that adding or removing comments (anywhere) or doc attributes, lint attributes etc. (outside of macro invocations) wouldn't change the build, if we're to do anything useful with this. (Plus, we'd need a new hash function.)

Though comments are most likely to contain metadata that should logically be mutable, so maybe we don't even need to go that far.

[Diggsey]

Everything downloaded from crates.io as part of a build must be immutable, even if it doesn't affect the final result. I would not use a registry that does anything else.

It's fine for there to be additional files/metadata that are "mutable" (although mutability is a lie anyway...) but that should not be downloaded when I run cargo build on a downstream crate.

When you publish a crate you are publishing a snapshot for people to depend on. You don't get to later change that snapshot, at all, that's what "publishing" means. If you publish a book, and make a mistake, you can't just locate all the copies and update them (and legally you would have no right to). All you can do is publish a new revised copy.

[wizzwizz4]

Does Cargo necessarily have to download the original source code for dependencies? I can imagine a cargo vendor --no-doc that doesn't download the info required for documentation, which would save on bandwidth for bandwidth-restricted users, build servers etc.. (Of course, we'd need some way to verify that we were actually getting an unmodified copy of the program, necessitating an additional stripped AST hash (or something) that would vary as the AST-stripping algorithm changed, so perhaps this isn't worth it.)

The main reason I raised this is that many licenses recommend adding a snippet of license text at the top of the source file. That, and other practices of embedding metadata in source files, puts a limit on how much we can do this.

Is the documentation logically part of the program? I could make arguments either way – consistency with README.md versus the argument-from-literate-programming.

[VulpineAmethyst]

Everything downloaded from crates.io as part of a build must be immutable, even if it doesn't affect the final result. I would not use a registry that does anything else.

It's fine for there to be additional files/metadata that are "mutable" (although mutability is a lie anyway...) but that should not be downloaded when I run cargo build on a downstream crate.

When you publish a crate you are publishing a snapshot for people to depend on. You don't get to later change that snapshot, at all, that's what "publishing" means. If you publish a book, and make a mistake, you can't just locate all the copies and update them (and legally you would have no right to). All you can do is publish a new revised copy.

Digital distribution is different from physical distribution in some pretty fundamental ways, in that you can make emendations at any time so that all future distributions of the product reflect those changes. And it's absurd to assert that changing ancillary metadata necessitates a new publication when there are no substantial changes to the work. What I'm proposing, in effect, is that crate-wide metadata be stapled to the package when it is downloaded, rather than requiring packages to contain a fixed version of that metadata that is then immutable.

[Diggsey]

What I'm proposing, in effect, is that crate-wide metadata be stapled to the package when it is downloaded, rather than requiring packages to contain a fixed version of that metadata that is then immutable.

I understand what you're proposing. I'm saying that, as a consumer of a crate, I did not agree to download your new ancillary metadata. When I add a dependency to my lockfile, I'm choosing to add it as it was. You don't get to decide in future that actually I wanted something else, even if the change should not affect the output of the build.

crates.io serves both crate publishers and crate consumers, it doesn't just serve the publishers.