Support PGO in the compiler's build system

[michaelwoerister]

One of the first steps in bringing PGO to rustc is adding support for it to the compiler's build system. Concretely this means

• support building an instrumented compiler (-Cprofile-generate phase), and
• support building a compiler that makes use of profile data (-Cprofile-use phase)

Optionally, the build system could also

• support profile data collection (by running the instrumented compiler through a standard set of workloads)

In rust-lang/cargo#7618 (comment), @luser suggests that the three phases should be operable separately. I think that makes a lot of sense since on CI we'll probably want to either only run the "profile-use" phase with downloaded profile data, or only the "profile-generate" phase that uploads the profile data.

I suggest adding the following new config.toml settings:

• llvm.profile-generate = ( "<path>" | "default-path" )
• llvm.profile-use = ( "<path>" | "default-path" | "download-profdata" )
• rust.profile-generate = ( "<path>" | "default-path" )
• rust.profile-use = ( "<path>" | "default-path" | "download-profdata" )
• rust.profile-collect = (true | false)

The paths specify the argument to the -Cprofile-generate and -Cprofile-use flags. default-path means some known location within the build directory. download-profdata means that the build system will try to find and download the most recent profile data available for the given commit. If rust.profile-collect is true, the build system will run a standard data collection scenario with the instrumented compiler (or will error if none of profile-generate flags is set).

Note that this setup has the following implications:

• One can turn PGO on or off independently for LLVM and Rust.
• There is no single command that runs all three phases at once. If one wants to build a PGOed compiler completely locally, one has to run something like:

  ./configure 
    --llvm.profile-generate=default-path
    --rust.profile-generate=default-path
    --llvm.profile-collect=true
  
  ./x.py build
  
  ./x.py clean # including LLVM?
  
  ./configure 
    --llvm.profile-use=default-path
    --rust.profile-use=default-path
  
  ./x.py build

• The "default" scenario where "generate" and "use" phases are not run on the same machine can be done with a single ./configure && ./x.py build though.

I'd like to hear if somebody can come up with a different, better setup. Maybe something that allows the entire three-phase build with a single ./x.py pgo-build command?

[Mark-Simulacrum]

I don't think we need to optimize for full collection + use, I'm not even sure how many people would use it if available -- it's just not really a common thing I expect.

One question is that for the Rust part of the PGO, I imagine we don't want to collect or use the PGO data for the first round of compilation, right? Since that'll have differing codegen from the second round (potentially even using a different LLVM version). That's fine -- rustbuild can handle that for us -- but something to keep in mind.

If we don't support simultaneously using and generating PGO data we should probably check that at startup but I'm not otherwise all that worried.

[michaelwoerister]

One question is that for the Rust part of the PGO, I imagine we don't want to collect or use the PGO data for the first round of compilation, right? Since that'll have differing codegen from the second round (potentially even using a different LLVM version). That's fine -- rustbuild can handle that for us -- but something to keep in mind.

Hm, I wonder if it would be possible to make stage 1 instrumented, use it to collect profile data, and then make stage 2 use that data. The profdata format looks rather robust, so that might even work for different LLVM versions. But I would not want to rely on it.

Are our release artifacts stage 2 builds? Then the instrumented compiler should also be a stage 2 build.

[Mark-Simulacrum]

Right, stage 2 is basically the artifacts we'd be using for this, though they're built as "stage 1 artifacts" -- the second rustc binary that gets produced. I'm not too worried about getting the language precise here, I think we have agreement on what we need to profile :)

I expect that we can build stage 0 artifacts as normal -- no need for special PGO handling, though eventually the beta they're built with will be PGO'd so that should give a nice speed up. Then the second round of compilation (stage 1 artifacts) is built, instrumented, we run our workloads to gather data, and then rebuild the second round of compilation using that data. This gives us a single compiler rebuild in terms of overhead.

Thinking along those lines, perhaps the right thing is actually for rustbuild to always do this - i.e., the pgo-build option you suggest is not actually necessary. Specifically, rustbuild will collect data if the profile-generate options are given and use it with profile-use; just profile-use will only use it, and both will do the full thing.

[luser]

Hm, I wonder if it would be possible to make stage 1 instrumented, use it to collect profile data, and then make stage 2 use that data. The profdata format looks rather robust, so that might even work for different LLVM versions. But I would not want to rely on it.

If this works on a technical level with the profile data format then this sounds like a pretty great idea. You've already got a multi-phase build, so reusing one of those phases to generate the profile data would be a clever trick! Per my comment elsewhere, this would impact the reproducibility of the builds. To preserve that it would be nice to publish the profile data as a build artifact and provide rustbuild support for producing an optimized build given an existing set of profile data.

[michaelwoerister]

I expect that we can build stage 0 artifacts as normal -- no need for special PGO handling, though eventually the beta they're built with will be PGO'd so that should give a nice speed up. Then the second round of compilation (stage 1 artifacts) is built, instrumented, we run our workloads to gather data, and then rebuild the second round of compilation using that data. This gives us a single compiler rebuild in terms of overhead.

Let me see if I understand this correctly: Right now we have:

beta --> stage0 --> stage1
                    ^^^^^^
                     dist

that would become:

beta --> stage0 --> stage1 --> stage2
                    ^^^^^^     ^^^^^^
                instrumented    PGO-ed dist

So, PGO would mean one additional build, right?

[michaelwoerister]

I'm wondering if that isn't actually too complicated since we'd actually never use that build in CI. What I suggested originally would look like:

# case 1: generate profile data

beta --> stage0 --> stage1 =run=> profdata (for uploading)
                    ^^^^^^
               dist (instrumented)


# case 2: use profile data (downloaded, from some previous build)

beta --> stage0 --> stage1
                    ^^^^^^
               dist (profile-use)

Cases (1) and (2) would always run in parallel, and case (2) would use profile data from a previous run of case (1). The advantage is that we don't have to worry about building LLVM twice.

[michaelwoerister]

Another wrinkle here that I just realized is that LLVM is built only once -- so for the profile-generate case we would work with an instrumented LLVM already when building the instrumented rustc. That should not be much of a problem though since instrumentation can be disabled at runtime via an environment variable, if I remember correctly. Or we just collect profile data for LLVM twice. The corresponding .profraw are automatically updated in place by the instrumentation runtime.