Description
This is a follow up to our discussion in #710 (comment) (and also referenced in #301, #179, and #524.
Now that location info is plumbed through the tokenizer, we can leverage these locations to:
- Produce better error messages in the parser (by referencing the offending token, and its location)
- Propagate location information outside of the parser, so that libraries that consume the parse tree can reference the original location of an AST node.
From my understanding, the first task should be easier, since we wouldn't need to update the parse tree data structure (perhaps outside of a few edge cases). The main change should be updating the parser to use TokenWithLocation and then improving the ParserError messages to reference these locations. I would suggest changing the type of TokenizerError and ParserError to be richer than a string, and pass along the location information directly, so that consumers of the error can access it directly.
The second task will be more difficult, however, since we'll need to either change all of the tree nodes to be structs instead of enums (to host the location information), or change each enum branch to be a struct with location information. I think the second approach will likely be easier to change incrementally, and for users to refactor against, and we can add a trait that homogenizes how users of the tree access the location info.
@alamb let me know your thoughts on the above.