Add syntax sources directly to compiler repo

Author: cknitt

.gitignore

@@ -55,7 +55,6 @@ coverage
 ppx_tools/
 vendor/ocaml/stdlib/target_camlheader
 vendor/ocaml/stdlib/target_camlheaderd
-reason/
 man/
 lib/ocaml
 *.tgz

res_syntax/.gitignore

@@ -0,0 +1,19 @@
+.DS_STORE
+*.cm*
+*.o
+*.s
+lib/*
+!lib/README.md
+.vscode/settings.json
+
+# Dune
+_build
+
+# Merlin
+.merlin
+
+# Local opam switch
+_opam/
+
+# Output dir for bootstrap script
+bootstrap/

res_syntax/.ocamlformat

@@ -0,0 +1,11 @@
+profile = default
+version = 0.22.4
+
+field-space = tight-decl
+break-cases = toplevel
+module-item-spacing = preserve
+cases-exp-indent = 2
+space-around-arrays = false
+space-around-lists = false
+space-around-records = false
+space-around-variants = false

res_syntax/.ocamlformat-ignore

@@ -0,0 +1 @@
+compiler-libs-406/*

res_syntax/CHANGELOG.md

@@ -0,0 +1,98 @@
+## Master
+
+> **Tags:**
+>
+> - :boom: [Breaking Change]
+> - :eyeglasses: [Spec Compliance]
+> - :rocket: [New Feature]
+> - :bug: [Bug Fix]
+> - :memo: [Documentation]
+> - :house: [Internal]
+> - :nail_care: [Polish]
+
+#### :boom: Breaking Change
+
+- Emit an error when a `@string` or `@int` attribute is used in a V4 component https://github.com/rescript-lang/rescript-compiler/issues/5724
+
+#### :rocket: New Feature
+
+- Add surface syntax for `async`/`await` https://github.com/rescript-lang/syntax/pull/600
+
+- Initial support for JSX V4, still work in progress.
+
+  - :boom: when V4 is activated, at most one component is allowed for each module.
+
+- Add support for empty record literal `{}` for records with only optional fields, and type definition of empty record (e.g. `type empty = {}`) https://github.com/rescript-lang/syntax/pull/632
+
+- Support the use of spread anywhere in list creation (e.g. `list{...x, 1, ...y, ...z}). https://github.com/rescript-lang/syntax/pull/692
+
+- Add support for the argument of `@react.component` to set a props type from the outside. https://github.com/rescript-lang/syntax/pull/699
+
+#### :bug: Bug Fix
+
+- Fix issue in formatting JSX spread props https://github.com/rescript-lang/syntax/pull/644
+- Fix pretty printer where it would print doc comments on the same line as other attributes https://github.com/rescript-lang/syntax/pull/642
+- Fix location issue in error messages with JSX V4 where the body of the component is an application https://github.com/rescript-lang/syntax/pull/633
+- Fix issue where the printer would omit attributes for `->` and `|>` https://github.com/rescript-lang/syntax/pull/629
+- Fix printing of optional fields in records https://github.com/rescript-lang/rescript-compiler/issues/5654
+- Fix printing of comments inside empty blocks https://github.com/rescript-lang/syntax/pull/647
+- Fix location issue in error messages with JSX V4 where the multiple props types are defined https://github.com/rescript-lang/syntax/pull/655
+- Fix location issue in make function in JSX V4 that breaks dead code elimination https://github.com/rescript-lang/syntax/pull/660
+- Fix parsing (hence pretty printing) of expressions with underscore `_` and comments.
+- Fix printing of comments inside JSX tag https://github.com/rescript-lang/syntax/pull/664
+- Fix issue where formatter erases tail comments inside JSX tag https://github.com/rescript-lang/syntax/issues/663
+- Fix issue where the JSX prop has type annotation of the first class module https://github.com/rescript-lang/syntax/pull/666
+- Fix issue where a spread `...x` in non-last position would not be reported as syntax error https://github.com/rescript-lang/syntax/pull/673/
+- Fix issue where the formatter would delete `async` in a function with labelled arguments.
+- Fix several printing issues with `async` including an infinite loop https://github.com/rescript-lang/syntax/pull/680
+- Fix issue where certain JSX expressions would be formatted differenctly in compiler 10.1.0-rc.1 https://github.com/rescript-lang/syntax/issues/675
+- Fix issue where printing nested pipe discards await https://github.com/rescript-lang/syntax/issues/687
+- Fix issue where the JSX key type is not an optional string https://github.com/rescript-lang/syntax/pull/693
+- Fix issue where the JSX fragment without children build error https://github.com/rescript-lang/syntax/pull/704
+- Fix issue where async as an id cannot be used with application and labelled arguments https://github.com/rescript-lang/syntax/issues/707
+- Treat await as almost-unary operator weaker than pipe so `await foo->bar` means `await (foo->bar)` https://github.com/rescript-lang/syntax/pull/711
+
+#### :eyeglasses: Spec Compliance
+
+- Functions with consecutive dots now print as multiple arrow functions like in JavaScript.
+
+#### :nail_care Polish
+
+- Change the internal representation of props for the lowercase components to record. https://github.com/rescript-lang/syntax/pull/665
+- Change the payload of Pconst_char for type safety. https://github.com/rescript-lang/rescript-compiler/pull/5759
+- Specialize the printing of the rhs of a record field assignment for optional values `{x: ? e}` https://github.com/rescript-lang/syntax/issues/714
+
+## ReScript 10.0
+
+- Fix printing for inline nullary functor types [#477](https://github.com/rescript-lang/syntax/pull/477)
+- Fix stripping of quotes for empty poly variants [#474](https://github.com/rescript-lang/syntax/pull/474)
+- Implement syntax for arity zero vs arity one in uncurried application in [#139](https://github.com/rescript-lang/syntax/pull/139)
+- Fix parsing of first class module exprs as part of binary/ternary expr in [#256](https://github.com/rescript-lang/syntax/pull/256)
+- Fix formatter hanging on deeply nested function calls [#261](https://github.com/rescript-lang/syntax/issues/261)
+- Remove parsing of "import" and "export" which was never officially supported.
+
+## ReScript 9.0.0
+
+- Fix parsing of poly-var typexpr consisting of one tag-spec-first in [#254](https://github.com/rescript-lang/syntax/pull/254)
+- Implement new syntax for guards on pattern match cases in [#248](https://github.com/rescript-lang/syntax/pull/248)
+- Implement intelligent breaking for poly-var type expressions in [#246](https://github.com/rescript-lang/syntax/pull/246)
+- Improve indentation of fast pipe chain in let binding in [#244](https://github.com/rescript-lang/syntax/pull/244)
+- Improve printing of non-callback arguments in call expressions with callback in [#241](https://github.com/rescript-lang/syntax/pull/241/files)
+- Fix printing of constrained expressions in rhs of js objects [#240](https://github.com/rescript-lang/syntax/pull/240)
+- Improve printing of trailing comments under lhs of "pipe" expression in [#329](https://github.com/rescript-lang/syntax/pull/239/files)
+- Improve printing of jsx children and props with leading line comment in [#236](https://github.com/rescript-lang/syntax/pull/236)
+- Improve conversion of quoted strings from Reason in [#238](https://github.com/rescript-lang/syntax/pull/238)
+- Print attributes/extension without bs prefix where possible in [#230](https://github.com/rescript-lang/syntax/pull/230)
+- Cleanup gentype attribute printing [fe05e1051aa94b16f6993ddc5ba9651f89e86907](https://github.com/rescript-lang/syntax/commit/fe05e1051aa94b16f6993ddc5ba9651f89e86907)
+- Implement light weight syntax for poly-variants [f84c5760b3f743f65e934195c87fc06bf88bff75](https://github.com/rescript-lang/syntax/commit/f84c5760b3f743f65e934195c87fc06bf88bff75)
+- Fix bug in fast pipe conversion from Reason. [3d5f2daba5418b821c577ba03e2de1afb0dd66de](https://github.com/rescript-lang/syntax/commit/3d5f2daba5418b821c577ba03e2de1afb0dd66de)
+- Improve parsed AST when tilde is missing in arrow expr parameters. [e52a0c89ac39b578a2062ef15fae2be625962e1f](https://github.com/rescript-lang/syntax/commit/e52a0c89ac39b578a2062ef15fae2be625962e1f)
+- Improve parser diagnostics for missing tilde in labeled parameters. [a0d7689d5d2bfc31dc251e966ac33a3001200171](https://github.com/rescript-lang/syntax/commit/a0d7689d5d2bfc31dc251e966ac33a3001200171)
+- Improve printing of uncurried application with a huggable expression in [c8767215186982e171fe9f9101d518150a65f0d7](https://github.com/rescript-lang/syntax/commit/c8767215186982e171fe9f9101d518150a65f0d7)
+- Improve printing of uncurried arrow typexpr outcome printer in [4d953b668cf47358deccb8b730566f24de25b9ee](https://github.com/rescript-lang/syntax/commit/4d953b668cf47358deccb8b730566f24de25b9ee)
+- Remove support for nativeint syntax in [72d9b7034fc28f317672c94994b322bee520acca](https://github.com/rescript-lang/syntax/commit/72d9b7034fc28f317672c94994b322bee520acca)
+- Improve printing of poly variant typexprs with attributes in [bf6561b](https://github.com/rescript-lang/syntax/commit/bf6561bb5d84557b8b6cbbcd40078c39526af4af)
+
+## ReScript 8.4.2 (December 11, 2020)
+
+Released in https://github.com/rescript-lang/syntax/releases/tag/v8.4.2 as part of https://github.com/rescript-lang/rescript-compiler/releases/tag/8.4.2

res_syntax/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 ReScript
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

res_syntax/Makefile

@@ -0,0 +1,36 @@
+SHELL = /bin/bash
+
+build:
+	dune build
+
+bootstrap: build
+	dune exec -- bash ./scripts/bootstrap.sh
+
+bench: build
+	dune exec -- bench
+
+test: build
+	dune exec -- testrunner
+	dune exec -- bash ./scripts/test.sh
+	make reanalyze
+	bash ./scripts/testok.sh
+
+roundtrip-test: build
+	dune exec -- testrunner
+	ROUNDTRIP_TEST=1 dune exec -- bash ./scripts/test.sh
+	make reanalyze
+	bash ./scripts/testok.sh
+
+reanalyze: build
+	reanalyze.exe -set-exit-code -all-cmt _build/default -suppress testrunner,compiler-libs-406 -exclude-paths compiler-libs-406 
+
+format:
+	dune build @fmt --auto-promote
+
+checkformat:
+	dune build @fmt
+
+clean:
+	dune clean
+
+.PHONY: bench clean test roundtrip-test reanalyze bootstrap build-native

res_syntax/README.md

@@ -0,0 +1,125 @@
+# ReScript Syntax ![Tests](https://github.com/rescript-lang/syntax/workflows/CI/badge.svg)
+
+Documentation: https://rescript-lang.org/docs/manual/latest/overview
+
+This repo is the source of truth for the ReScript parser & printer. Issues go here.
+
+**You don't need this repo to use the ReScript syntax**. This comes with ReScript >=8.1. This repo is for syntax developers.
+
+## Contribute
+
+### Why
+
+A detailed discussion by Jonathan Blow and Casey Muratori on why you would hand-roll a parser for a production quality programming language
+[Discussion: Making Programming Language Parsers, etc](https://youtu.be/MnctEW1oL-E)
+
+"One reason why I switched off these parser tools is that the promises didn't really materialize.
+The amount of work that I had to do change a yacc script from one language to a variant of that language
+was more than if I hand wrote the code myself.
+"
+J. Blow.
+
+### Setup & Usage (For Repo Devs Only)
+
+Required:
+
+- OCaml 4.10 or later
+- Dune
+- Reanalyze
+- OS: macOS, Linux or Windows
+
+```sh
+git clone https://github.com/rescript-lang/syntax.git
+cd syntax
+opam install . --deps-only --with-test
+make # or "dune build"
+```
+
+This will produce the three binaries `rescript`, `tests` and `bench` (with `.exe` extension on Windows).
+
+We only build production binaries, even in dev mode. No need for a separate dev binary when the build is fast enough. Plus, this encourages proper benchmarking of the (production) binary each diff.
+
+After you make a change:
+
+```sh
+make
+```
+
+Run the core tests:
+
+```sh
+make test
+```
+
+Run the extended tests (not fully working on Windows yet):
+
+```sh
+make roundtrip-test
+```
+
+Those will tell you whether you've got a test output difference. If it's intentional, check them in.
+
+Debug a file:
+
+```sh
+# write code in test.res
+dune exec -- rescript test.res # test printer
+dune exec -- rescript -print ast test.res # print ast
+dune exec -- rescript -print comments test.res # print comment table
+dune exec -- rescript -print ml test.res # show ocaml code
+dune exec -- rescript -print res -width 80 test.res # test printer and change default print width
+```
+
+Benchmark:
+
+```sh
+make bench
+```
+
+Enable stack trace:
+
+```sh
+# Before you run the binary
+export OCAMLRUNPARAM="b"
+```
+
+This is likely a known knowledge: add the above line into your shell rc file so that every shell startup you have OCaml stack trace enabled.
+
+### Development Docs
+
+#### Folder Structure
+
+- `src` contains all the parser/printer source code. Don't change folder structure without notice; The [rescript-compiler](https://github.com/rescript-lang/rescript-compiler) repo uses this repo as a submodule and assumes `src`.
+- `benchmarks`, `cli` and `tests` contain the source code for the executables used for testing/benchmarking. These are not used by the [rescript-compiler](https://github.com/rescript-lang/rescript-compiler) repo.
+
+#### Error Reporting Logic
+
+Right now, ReScript's compiler's error reporting mechanism, for architectural reasons, is independent from this syntax repo's error reporting mechanism. However, we do want a unified look when they report the errors in the terminal. This is currently achieved by (carefully...) duplicating the error report logic from the compiler repo to here (or vice-versa; either way, just keep them in sync). The files to sync are the compiler repo's [super_location.ml](https://github.com/rescript-lang/rescript-compiler/blob/fcb21790dfb0592f609818df7790192061360631/jscomp/super_errors/super_location.ml) and [super_code_frame.ml](https://github.com/rescript-lang/rescript-compiler/blob/fcb21790dfb0592f609818df7790192061360631/jscomp/super_errors/super_code_frame.ml), into this repo's [res_diagnostics_printing_utils.ml](https://github.com/rescript-lang/syntax/blob/ec5cefb23b659b0a7be170ae0ad26f3fe8a05456/src/res_diagnostics_printing_utils.ml). A few notes:
+
+- Some lines are lightly changed to fit this repo's needs; they're documented in the latter file.
+- Please keep these files lightweight and as dependency-less as possible, for easier syncing.
+- The syntax logic currently doesn't have warnings, only errors, and potentially more than one.
+- In the future, ideally, the error reporting logic would also be unified with GenType and Reanalyze's. It'd be painful to copy paste around all this reporting logic.
+- The errors are reported by the parser [here](https://github.com/rescript-lang/syntax/blob/ec5cefb23b659b0a7be170ae0ad26f3fe8a05456/src/res_diagnostics.ml#L146).
+- Our editor plugin parses the error report from the compiler and from the syntax [here](https://github.com/rescript-lang/rescript-vscode/blob/0dbf2eb9cdb0bd6d95be1aee88b73830feecb5cc/server/src/utils.ts#L129-L329).
+
+### Example API usage
+
+```ocaml
+let filename = "foo.res"
+let src = FS.readFile filename
+
+let p =
+  (* intended for ocaml compiler *)
+  let mode = Res_parser.ParseForTypeChecker in
+  (* if you want to target the printer use: let mode = Res_parser.Default in*)
+  Res_parser.make ~mode src filename
+
+let structure = Res_core.parseImplementation p
+let signature = Res_core.parseSpecification p
+
+let () = match p.diagnostics with
+| [] -> () (* no problems *)
+| diagnostics -> (* parser contains problems *)
+  Res_diagnostics.printReport diagnostics src
+```