expand notes on expansion heirarchies

Author: mark-i-m

src/macro-expansion.md

@@ -163,82 +163,153 @@ only within the macro (i.e. it should not be visible outside the macro).
 [code_parse_int]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/mbe/macro_parser/fn.parse_tt.html
 [parsing]: ./the-parser.html
 
-TODO: expand these notes
+The context is attached to AST nodes. All AST nodes generated by macros have
+context attached. Additionally, there may be other nodes that have context
+attached, such as some desugared syntax (non-macro-expanded nodes are
+considered to just have the "root" context, as described below).
 
-- Many AST nodes have some sort of syntax context, especially nodes from macros.
-- When we ask what is the syntax context of a node, the answer actually differs by what we are trying to do. Thus, we don't just keep track of a single context. There are in fact 3 different types of context used for different things.
-- Each type of context is tracked by an "expansion heirarchy". As we expand macros, new macro calls or macro definitions may be generated, leading to some nesting. This nesting is where the heirarchies come from. Each heirarchy tracks some different aspect, though, as we will see.
-- There are 3 expansion heirarchies
-    - All macros receive an integer ID assigned continuously starting from 0 as we discover new macro calls
-        - This is used as the `expn_id` where needed.
-    - All heirarchies start at ExpnId::root, which is its own parent
-    - The context of a node consists of a chain of expansions leading to `ExpnId::root`. A non-macro-expanded node has syntax context 0 (`SyntaxContext::empty()`) which represents just the root node.
-    - There are vectors in `HygieneData` that contain expansion info.
-        - There are entries here for both `SyntaxContext::empty()` and `ExpnId::root`, but they aren't used much.
+Because macros invocations and definitions can be nested, the syntax context of
+a node must be a heirarchy. For example, if we expand a macro and there is
+another macro invocation or definition in the generated output, then the syntax
+context should reflex the nesting.
 
-    1. Tracks expansion order: when a macro invocation is in the output of another macro.
-        ...
-            expn_id2
-                expn_id1
-        InternalExpnData::parent is the child->parent link. That is the expn_id1 points to expn_id2 points to ...
+However, it turns out that there are actually a few types of context we may
+want to track for different purposes. Thus, there not just one but _three_
+expansion heirarchies that together comprise the hygiene information for a
+crate.
 
-        Ex:
-            macro_rules! foo { () => { println!(); } }
-            fn main() { foo!(); }
+All of these heirarchies need some sort of "macro ID" to identify individual
+elements in the chain of expansions. This ID is [`ExpnId`].  All macros receive
+an integer ID, assigned continuously starting from 0 as we discover new macro
+calls.  All heirarchies start at [`ExpnId::root()`][rootid], which is its own
+parent.
 
-            // Then AST nodes that are finally generated would have parent(expn_id_println) -> parent(expn_id_foo), right?
+The actual heirarchies are stored in [`HygieneData`][hd], and all of the
+hygiene-related algorithms are implemented in [`rustc_span::hygiene`][hy], with
+the exception of some hacks [`Resolver::resolve_crate_root`][hacks].
 
-    2. Tracks macro definitions: when we are expanding one macro another macro definition is revealed in its output.
-        ...
-            SyntaxContext2
-                SyntaxContext1
-        SyntaxContextData::parent is the child->parent link here.
-        SyntaxContext is the whole chain in this hierarchy, and SyntaxContextData::outer_expns are individual elements in the chain.
+[`ExpnId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnId.html
+[rootid]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnId.html#method.root
+[hd]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.HygieneData.html
+[hy]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/index.html
+[hacks]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Resolver.html#method.resolve_crate_root
 
-        - For built-in macros (e.g. `line!()`) or stable proc macros: tokens produced by the macro are given the context `SyntaxContext::empty().apply_mark(expn_id)`
-            - Such macros are considered to have been defined at the root.
-                - For proc macros this is because they are always cross-crate and we don't have cross-crate hygiene implemented.
+### The Expansion Order Heirarchy
 
-        The second hierarchy has the context transplantation hack. See https://github.com/rust-lang/rust/pull/51762#issuecomment-401400732.
+The first heirarchy tracks the order of expansions, i.e., when a macro
+invocation is in the output of another macro.
 
-        If the token had context X before being produced by a macro then after being produced by the macro it has context X -> macro_id.
+Here, the children in the heirarchy will be the "innermost" tokens.
+[`ExpnData::parent`][edp] tracks the child -> parent link in this heirarchy.
 
-        Ex:
-            ```rust
-            macro m() { ident }
-            ```
+[edp]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnData.html#structfield.parent
 
-            Here `ident` originally has context SyntaxContext::root(). `ident` has context ROOT -> id(m) after it's produced by m.
-            The "chaining operator" is `apply_mark` in compiler code.
+For example,
 
-        Ex:
+```rust,ignore
+macro_rules! foo { () => { println!(); } }
+
+fn main() { foo!(); }
+```
+
+In this code, the AST nodes that are finally generated would have heirarchy:
+
+```
+root
+    expn_id_foo
+        expn_id_println
+```
+
+### The Macro Definition Heirarchy
+
+The second heirarchy tracks the order of macro definitions, i.e., when we are
+expanding one macro another macro definition is revealed in its output.  This
+one is a bit tricky and more complex than the other two heirarchies.
+
+Here, [`SyntaxContextData::parent`][scdp] is the child -> parent link here.
+[`SyntaxContext`][sc] is the whole chain in this hierarchy, and
+[`SyntaxContextData::outer_expns`][scdoe] are individual elements in the chain.
+The "chaining operator" is [`SyntaxContext::apply_mark`][am] in compiler code.
+
+[scdp]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContextData.html#structfield.parent
+[sc]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html
+[scdoe]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContextData.html#structfield.outer_expn
+[am]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html#method.apply_mark
+
+For built-in macros, we use the context:
+`SyntaxContext::empty().apply_mark(expn_id)`, and such macros are considered to
+be defined at the heirarchy root. We do the same for proc-macros because we
+haven't implemented cross-crate hygiene yet.
+
+If the token had context `X` before being produced by a macro then after being
+produced by the macro it has context `X -> macro_id`. Here are some examples:
+
+Example 0:
+
+```rust,ignore
+macro m() { ident }
+
+m!();
+```
+
+Here `ident` originally has context [`SyntaxContext::root()`][scr]. `ident` has
+context `ROOT -> id(m)` after it's produced by `m`.
+
+[scr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html#method.root
 
-            ```rust
-            macro m() { macro n() { ident } }
-            ```
-            In this example the ident has context ROOT originally, then ROOT -> id(m), then ROOT -> id(m) -> id(n).
 
-        Note that these chains are not entirely determined by their last element, in other words ExpnId is not isomorphic to SyntaxCtxt.
+Example 1:
 
-        Ex:
-            ```rust
-            macro m($i: ident) { macro n() { ($i, bar) } }
+```rust,ignore
+macro m() { macro n() { ident } }
+
+m!();
+n!();
+```
+In this example the `ident` has context `ROOT` originally, then `ROOT -> id(m)`
+after the first expansion, then `ROOT -> id(m) -> id(n)`.
+
+Example 2:
+
+Note that these chains are not entirely determined by their last element, in
+other words `ExpnId` is not isomorphic to `SyntaxContext`.
+
+```rust,ignore
+macro m($i: ident) { macro n() { ($i, bar) } }
+
+m!(foo);
+```
+
+After all expansions, `foo` has context `ROOT -> id(n)` and `bar` has context
+`ROOT -> id(m) -> id(n)`.
 
-            m!(foo);
-            ```
+Finally, one last thing to mention is that currently, this heirarchy is subject
+to the ["context transplantation hack"][hack]. Basically, the more modern (and
+experimental) `macro` macros have stronger hygiene than the older MBE system,
+but this can result in weird interactions between the two. The hack is intended
+to make things "just work" for now.
 
-            After all expansions, foo has context ROOT -> id(n) and bar has context ROOT -> id(m) -> id(n)
+[hack]: https://github.com/rust-lang/rust/pull/51762#issuecomment-401400732
 
-    3. Call-site: tracks the location of the macro invocation.
-        Ex:
-            If foo!(bar!(ident)) expands into ident
-            then hierarchy 1 is root -> foo -> bar -> ident
-            but hierarchy 3 is root -> ident
+### The Call-site Heirarchy
 
-        ExpnInfo::call_site is the child-parent link in this case.
+The third and final heirarchy tracks the location of macro invocations.
+
+In this heirarchy [`ExpnData::call_site`][callsite] is the child -> parent link.
+
+[callsite]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnData.html#structfield.call_site
+
+Here is an example:
+
+```rust,ignore
+macro bar($i: ident) { $i }
+macro foo($i: ident) { $i }
+
+foo!(bar!(baz));
+```
 
-- Hygiene-related algorithms are entirely in hygiene.rs
-    - Some hacks in `resolve_crate_root`, though.
+For the `baz` AST node in the final output, the first heirarchy is `ROOT ->
+id(foo) -> id(bar) -> baz`, while the third heirarchy is `ROOT -> baz`.
 
 ## Producing Macro Output