RFC: Report on Implementations' Statuses

[Julian]

There should be a mechanism by which implementers (currently mentioned in the README or otherwise) can submit their implementation to be automatically tested and displayed on each commit to the test suite.

Specifically, we would report on:

• number of failing tests
• supported drafts

At first thought, an easy implementation may be to have each participating implementation implement a black-box executable which:

• takes one test as input on stdin, and one argument, the metaschema ID for the metaschema of the provided test
• exits successfully for valid instances
• exits with non-zero exit status for invalid ones
• optionally writes to stdout an object in the draft2019-09 (or later) output format containing output about the validation process

Internally then (here in the test repo), we run each of these binaries via a GitHub action, collect the results, and expose them for viewing.

(This issue is very open for discussion from implementers -- comments welcome.)

[Julian]

@gregsdennis found https://cburgmer.github.io/json-path-comparison/ for a similar effort on JSON path.

[gregsdennis]

I've invited him to do the same thing for us (or at the very least let us copy/modify what he's done).

cburgmer/json-path-comparison#27

[gregsdennis]

Looks like it's a copy/modify, if people here are okay with going that direction.

[gregsdennis]

@cburgmer wrote up some of the logic behind his test suite. It's a really good read and a lot of the reasoning applies well to JSON Path.

Primarily of note, there are no "expected" test results. He's truly performing a comparison between implementations and reporting on what he calls the consensus between them, determined by a majority-plus-one of the available implementations agreeing on the output. He does this because there is no specification for JSON Path, so no really "correct" response for any given query.

Since we have a specification, we don't need the consensus concept. Our test suite explicitly states the expected result, so we just need to report adherence to that result. The thing I like from his report is the report itself.

I'll work on modifying it so that it can at least run my implementation. I'd like other implementors to add their own libraries, but if I can work out how to do some of them, I'll start at the top of the implementation list on the site and work down.

[cburgmer]

If it helps you I can call out the complexity in the json-path-comparison:

1. Consensus building
2. Understanding the domain and define test cases
3. Handle implementations across all languages

As 3. probably applies to your goal as well I can clarify this point:
3.1. Provide one view into the different languages.
3.2. Make results reproducible (reduce source of indeterminism)
3.3. Deal with the sheer amount of work that's different for every tool stack.
3.4. Make it work across operating systems (Docker)

[Zac-HD]

It would be great to keep the consensus logic though, because this test suite isn't the only data we could compare implementations on.

If we have a blackbox setup for various implementations, I'd be happy to set up a fuzzing harness which uses hypothesis-jsonschema to generate random instances matching (or almost matching) both real-world or randomly generated schemas. It's been ridiculously effective for me, and I'd be interested in seeing the minimal examples about which implementations disagreed...