design discussion: build queue overhaul

[QuietMisdreavus]

Consider the following situations:

• The nightly compiler used by docs.rs to build docs is updated by hand. Sometimes the cadence of these updates means some crates will fail to build until the nightly is updated.
• Rustdoc occasionally introduces bugs that materially affect some aspect of the docs, so when docs.rs uses a rustdoc with these bugs, they reflect on any crate released or rebuilt while that version was active.
• The visual style of rustdoc output has changed over time, causing some docs to look different than others because they were built with sufficiently old versions of rustdoc.

I feel like there are two factors that contribute to these situations:

• The rustdoc used in the build container is updated by hand.
• Once a crate is released, the only way for that version's docs to get rebuilt is by hand.

The first of these is not something i'm willing to address right now - we recently had a period where docs.rs required code changes due to changes in rustdoc, and i'm not convinced we can completely automatically update our nightly without changes to rustdoc that would allow docs.rs to handle these changes better.

However, i believe we can address the second one without too much change to the architecture of docs.rs, by augmenting the build queue. Currently, the build queue is only operated by the thread that checks for new releases, and nothing else adds or removes items from it. I think we can extend the build queue by adding a "priority" to it, and letting the builder thread pull crates out according to this priority instead of just by date.

Here's my basic idea:

1. Break the current build queue thread into two: "check for new releases" and "build crates from the queue". Doing this gives us more leeway to add things to the queue from other sources - for example, we could add a command to the cratesfyi binary that adds a new line to the queue so we can use that instead of running the build command directly.
2. Add a priority column to the queue table in the database, and make the builder thread sort entries by priority as well as date. This enables us to add build requests from other sources without choking out new releases, especially if we change the build queue thread to only take one row from the build queue at a time, rather than the entire table at once.
3. Allow users to request a rebuild for their crate/version from the site, at a lower priority than new releases. This is a feature that has been requested before, and using the priority like this allows it to be built when there are no new releases to build.
4. Eagerly queue up failed builds from old rustdoc versions when rustdoc is updated, at a lower priority than user requests. For the same reason as before, this lets us try to eagerly rebuild crates that may have failed due to the old rustdoc, while not clogging up the queue for new releases.
5. Add a command to the cratesfyi binary that queues up builds for all crates which were built with a specific version of rustdoc, at a lower priority than previously-failed builds. Even though this is a manual action, it allows us to proactively rebuild crates that have issues like the [src] page line numbers problem without manually rebuilding every crate that used that version (and again, without choking new releases out of the build queue).

The biggest problem with this plan is that i don't know if there's a way to automatically make a database schema change while updating the site. While there's a cratesfyi database init command to create all the tables, there's no such command to apply table changes to an existing database. I'm not sure that's a hard requirement, but it would make deploying the change to prod easier.

[jyn514]

It looks like 1 and 2 have been added since February.

I'm not sure how hard 3. would be to add - I think this would basically be adding a REST endpoint and a button for crate owners but I could be wrong.

4. and 5. sound very doable.

automatically make a database schema change

I think this was implemented in #305

[pietroalbini]

I don't want to add any kind of authentication to docs.rs until we stop running untrusted JS on the docs.rs domain.

[jyn514]

Ah right, good call. We could still requeue failed builds though (4 and 5).

[jyn514]

I don't want to add any kind of authentication to docs.rs until we stop running untrusted JS on the docs.rs domain.

Hmm, the only issue is with JavaScript, right? We could have a REST endpoint that doesn't go through the website, with just a tiny script you could build from source and provide your crates.io credentials to. We'd need to set up a way for crates.io to tell docs.rs things are authenticated without providing credentials to docs.rs itself, but it at least seems possible in theory.