1- # Bootstrapping Rust
1+ # rustbuild - Bootstrapping Rust
22
33This is an in-progress README which is targeted at helping to explain how Rust
44is bootstrapped and in general some of the technical details of the build
@@ -8,20 +8,64 @@ system.
88> intended to be the primarily used one just yet. The makefiles are currently
99> the ones that are still "guaranteed to work" as much as possible at least.
1010
11- ## Using the new build system
11+ ## Using rustbuild
1212
1313When configuring Rust via ` ./configure ` , pass the following to enable building
1414via this build system:
1515
1616```
1717./configure --enable-rustbuild
18+ make
1819```
1920
20- ## ...
21+ Afterwards the ` Makefile ` which is generated will have a few commands like
22+ ` make check ` , ` make tidy ` , etc. For finer-grained control, the
23+ ` bootstrap.py ` entry point can be used:
24+
25+ ```
26+ python src/bootstrap/bootstrap.py
27+ ```
28+
29+ This accepts a number of options like ` --stage ` and ` --step ` which can configure
30+ what's actually being done.
31+
32+ ## Configuring rustbuild
33+
34+ There are currently two primary methods for configuring the rustbuild build
35+ system. First, the ` ./configure ` options serialized in ` config.mk ` will be
36+ parsed and read. That is, if any ` ./configure ` options are passed, they'll be
37+ handled naturally.
38+
39+ Next, rustbuild offers a TOML-based configuration system with a ` config.toml `
40+ file in the same location as ` config.mk ` . An example of this configuration can
41+ be found at ` src/bootstrap/config.toml.example ` , and the configuration file
42+ can also be passed as ` --config path/to/config.toml ` if the build system is
43+ being invoked manually (via the python script).
44+
45+ ## Build stages
46+
47+ The rustbuild build system goes through a few phases to actually build the
48+ compiler. What actually happens when you invoke rustbuild is:
49+
50+ 1 . The entry point script, ` src/bootstrap/bootstrap.py ` is run. This script is
51+ responsible for downloading the stage0 compiler/Cargo binaries, and it then
52+ compiles the build system itself (this folder). Finally, it then invokes the
53+ actual ` boostrap ` binary build system.
54+ 2 . In Rust, ` bootstrap ` will slurp up all configuration, perform a number of
55+ sanity checks (compilers exist for example), and then start building the
56+ stage0 artifacts.
57+ 3 . The stage0 ` cargo ` downloaded earlier is used to build the standard library
58+ and the compiler, and then these binaries are then copied to the ` stage1 `
59+ directory. That compiler is then used to generate the stage1 artifacts which
60+ are then copied to the stage2 directory, and then finally the stage2
61+ artifacts are generated using that compiler.
62+
63+ The goal of each stage is to (a) leverage Cargo as much as possible and failing
64+ that (b) leverage Rust as much as possible!
2165
2266## Directory Layout
2367
24- This build system houses all output under the ` target ` directory, which looks
68+ This build system houses all output under the ` build ` directory, which looks
2569like this:
2670
2771```
@@ -42,6 +86,12 @@ build/
4286 debug/
4387 release/
4488
89+ # Output of the dist-related steps like dist-std, dist-rustc, and dist-docs
90+ dist/
91+
92+ # Temporary directory used for various input/output as part of various stages
93+ tmp/
94+
4595 # Each remaining directory is scoped by the "host" triple of compilation at
4696 # hand.
4797 x86_64-unknown-linux-gnu/
@@ -50,7 +100,8 @@ build/
50100 # folder is under. The exact layout here will likely depend on the platform,
51101 # and this is also built with CMake so the build system is also likely
52102 # different.
53- compiler-rt/build/
103+ compiler-rt/
104+ build/
54105
55106 # Output folder for LLVM if it is compiled for this target
56107 llvm/
@@ -67,6 +118,17 @@ build/
67118 share/
68119 ...
69120
121+ # Output folder for all documentation of this target. This is what's filled
122+ # in whenever the `doc` step is run.
123+ doc/
124+
125+ # Output for all compiletest-based test suites
126+ test/
127+ run-pass/
128+ compile-fail/
129+ debuginfo/
130+ ...
131+
70132 # Location where the stage0 Cargo and Rust compiler are unpacked. This
71133 # directory is purely an extracted and overlaid tarball of these two (done
72134 # by the bootstrapy python script). In theory the build system does not
@@ -82,7 +144,9 @@ build/
82144 # invocation. The build system instruments calling Cargo in the right order
83145 # with the right variables to ensure these are filled in correctly.
84146 stageN-std/
147+ stageN-test/
85148 stageN-rustc/
149+ stageN-tools/
86150
87151 # This is a special case of the above directories, **not** filled in via
88152 # Cargo but rather the build system itself. The stage0 compiler already has
@@ -96,7 +160,7 @@ build/
96160 # Basically this directory is just a temporary artifact use to configure the
97161 # stage0 compiler to ensure that the libstd we just built is used to
98162 # compile the stage1 compiler.
99- stage0-rustc /lib/
163+ stage0-sysroot /lib/
100164
101165 # These output directories are intended to be standalone working
102166 # implementations of the compiler (corresponding to each stage). The build
@@ -108,3 +172,69 @@ build/
108172 stage2/
109173 stage3/
110174```
175+
176+ ## Cargo projects
177+
178+ The current build is unfortunately not quite as simple as ` cargo build ` in a
179+ directory, but rather the compiler is split into three different Cargo projects:
180+
181+ * ` src/rustc/std_shim ` - a project which builds and compiles libstd
182+ * ` src/rustc/test_shim ` - a project which builds and compiles libtest
183+ * ` src/rustc ` - the actual compiler itself
184+
185+ Each "project" has a corresponding Cargo.lock file with all dependencies, and
186+ this means that building the compiler involves running Cargo three times. The
187+ structure here serves two goals:
188+
189+ 1 . Facilitating dependencies coming from crates.io. These dependencies don't
190+ depend on ` std ` , so libstd is a separate project compiled ahead of time
191+ before the actual compiler builds.
192+ 2 . Splitting "host artifacts" from "target artifacts". That is, when building
193+ code for an arbitrary target you don't need the entire compiler, but you'll
194+ end up needing libraries like libtest that depend on std but also want to use
195+ crates.io dependencies. Hence, libtest is split out as its own project that
196+ is sequenced after ` std ` but before ` rustc ` . This project is built for all
197+ targets.
198+
199+ There is some loss in build parallelism here because libtest can be compiled in
200+ parallel with a number of rustc artifacts, but in theory the loss isn't too bad!
201+
202+ ## Build tools
203+
204+ We've actually got quite a few tools that we use in the compiler's build system
205+ and for testing. To organize these, each tool is a project in ` src/tools ` with a
206+ corresponding ` Cargo.toml ` . All tools are compiled with Cargo (currently having
207+ independent ` Cargo.lock ` files) and do not currently explicitly depend on the
208+ compiler or standard library. Compiling each tool is sequenced after the
209+ appropriate libstd/libtest/librustc compile above.
210+
211+ ## Extending rustbuild
212+
213+ So you'd like to add a feature to the rustbuild build system or just fix a bug.
214+ Great! One of the major motivational factors for moving away from ` make ` is that
215+ Rust is in theory much easier to read, modify, and write. If you find anything
216+ excessively confusing, please open an issue on this and we'll try to get it
217+ documented or simplified pronto.
218+
219+ First up, you'll probably want to read over the documentation above as that'll
220+ give you a high level overview of what rustbuild is doing. You also probably
221+ want to play around a bit yourself by just getting it up and running before you
222+ dive too much into the actual build system itself.
223+
224+ After that, each module in rustbuild should have enough documentation to keep
225+ you up and running. Some general areas that you may be interested in modifying
226+ are:
227+
228+ * Adding a new build tool? Take a look at ` build/step.rs ` for examples of other
229+ tools, as well as ` build/mod.rs ` .
230+ * Adding a new compiler crate? Look no further! Adding crates can be done by
231+ adding a new directory with ` Cargo.toml ` followed by configuring all
232+ ` Cargo.toml ` files accordingly.
233+ * Adding a new dependency from crates.io? We're still working on that, so hold
234+ off on that for now.
235+ * Adding a new configuration option? Take a look at ` build/config.rs ` or perhaps
236+ ` build/flags.rs ` and then modify the build elsewhere to read that option.
237+ * Adding a sanity check? Take a look at ` build/sanity.rs ` .
238+
239+ If you have any questions feel free to reach out on ` #rust-internals ` on IRC or
240+ open an issue in the bug tracker!
0 commit comments