Edit lexing

Author: pierwill

src/overview.md

@@ -118,6 +118,25 @@ Command line argument parsing occurs in the [`rustc_driver`]. This crate
 defines the compile configuration that is requested by the user and passes it
 to the rest of the compilation process as a [`rustc_interface::Config`].
 
+### Lexing and parsing
+
+The raw Rust source text is analyzed by a low-level *lexer* located in
+[`rustc_lexer`]. At this stage, the source text is turned into a stream of
+atomic source code units known as _tokens_.  The lexer supports the
+Unicode character encoding.
+
+The token stream passes through a higher-level lexer located in
+[`rustc_parse`] to prepare for the next stage of the compile process. The
+[`StringReader`] struct is used at this stage to perform a set of validations
+and turn strings into interned symbols (_interning_ is discussed later).
+[String interning] is a way of storing only one immutable
+copy of each distinct string value.
+
+The lexer has a small interface and doesn't depend directly on the
+diagnostic infrastructure in `rustc`. Instead it provides diagnostics as plain
+data which are emitted in `rustc_parse::lexer::mod` as real diagnostics.
+The lexer preserves full fidelity information for both IDEs and proc macros.
+
 
 [String interning]: https://en.wikipedia.org/wiki/String_interning
 [`rustc_lexer`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lexer/index.html