Increase testing & SQL support

[Dandandan]

I want to write down some ideas about increasing support for commonly used SQL statement.

Some different ideas for doing more testing:

1. Add SQL corpora that should be parsed correctly. I think we can create a directory with multiple dialects with multiple corpora.
   Examples can be TPCDS benchmark http://www.tpc.org/tpcds/ and other resources from different projects.
   We can test two main things here:

• Whether everything parses to a statement
• When serialized (e.g. to JSON), whether it outputs the same thing as a earlier version of slqparser.

2. Increasing test coverage by adding unit tests
3. Increasing test coverage (not only when measuring coverage by line) by adding property-based tests https://github.com/BurntSushi/quickcheck https://github.com/AltSysrq/proptest
4. Add debug asserts, to make sure any assumptions about the inputs are correct. Because those are enabled during testing, they can be very helpful in general to make sure those really hold.
5. Fuzzing https://github.com/andygrove/sqlparser-rs/pull/211 . This can help with finding any edge cases in the parser which crashes without much effort.

Some great list of general testing strategies can be found here:
https://www.sqlite.org/testing.html

Any feedback, ideas on this?

[nickolay]

When serialized (e.g. to JSON), whether it outputs the same thing as a earlier version of slqparser.

Yeah, this is called "snapshot testing" and I think it could replace most of our current tests.
The main benefit to me is the simplicity of adding a new test, compared to the status quo, which can lead to increased coverage.

(I don't see the reason to use JSON here and like to avoid heavy-weight dependencies like serde for essential functionality, if possible, but that can be discussed separately.)

When I investigated this a year or so ago, I didn't like the existing tools like insta (for most of these reasons). I like the ergonomics of rust-analyzer's parser tests (docs, tests directory, tests entry point).

Increasing test coverage by adding unit tests

Not sure what you mean by "unit" tests, but I like the idea of inline tests in rust-analyzer, which serve as both code comments and a kind of unit-tests.

3. https://github.com/BurntSushi/quickcheck .. 4. debug asserts .. 5. Fuzzing Add fuzzer based on cargo-fuzz #211

I can't say I understand how specifically 3-5 would help. Perhaps demonstrating real bugs found in sqlparser-rs using those would help.

[Dandandan]

1. Yes, we could also use the displayed output instead of serde, that would be much better (and cover that displaying functionality / regressions a bit more as well).

2. With unit tests I mean the current tests that present a simple input and output.

3-4 main benefit is making the amount of test cases much larger compared to handwritten examples. For example, the fuzzer found some cases in which the parser is currently extremely slow #217 and could spot crashing code (if present). Quickcheck/property testing would allow to test some properties with a large number of randomized inputs.

5. Debug asserts can be useful to add assumptions about the state / input that should hold (about input, state).