Encountering `$schema` during meta-schema validation

[gregsdennis]

In #1434, @karenetheridge left a comment regarding special-casing meta-schema validations. This issue continues that conversation.

When validating a schema with a meta-schema that contains an subschema with a different $schema value, the meta-schema cannot be expected to validate the entire document. Rather it must only apply to the appropriate portions of the schema.

An example to illustrate the problem:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "base",
  "type": "object",
  "properties": {
    "foo": { "$ref": "foo" }
  },
  "$defs": {
    "foo-def": {
      "$schema": "http://json-schema.org/draft-07/schema#",
      "$id": "foo",
      "type": "array",
      "items": [
        { "type": "string" },
        { "type": "integer" }
      ]
    }
  }
}

This schema is valid, but it changes meta-schemas in the foo-def definition. Validating this as an instance against draft 2020-12 would fail because foo#/items is the array-form items which is disallowed in 2020-12.

It's necessary for the validator to have special knowledge that it's processing a meta-schema so that when it encounters a new $schema it can switch meta-schemas appropriately.

While Core 9.3.3 does address this, does it need to be clearer?

Given that a Compound Schema Document may have embedded resources which identify as using different dialects, these documents SHOULD NOT be validated by applying a meta-schema to the Compound Schema Document as an instance. It is RECOMMENDED that an alternate validation process be provided in order to validate Schema Documents. Each Schema Resource SHOULD be separately validated against its associated meta-schema.

This also ties in a little to Using $schema in JSON documents.

[jdesrosiers]

If someone has better words suggest that they think are more clear, I'm open to that, but I think the spec already says everything it needs to say.

[gregsdennis]

@Julian do you think we can set up a test for this? I mean, I created a test case above, but:

1. Can the test indicate that it should be validating as a meta-schema?
2. Is it a problem that this spans multiple drafts?

We already have the cross-draft.json tests in the optional directories, but they test changing drafts through a $ref rather than being embedded.

[jdesrosiers]

We already have the cross-draft.json tests in the optional directories, but they test changing drafts through a $ref rather than being embedded.

The whole point of redefining $id as indicating an embedded schema was to make the behavior identical to referencing an external schema. So, I think "cross-draft" is a perfect place for these kinds of tests.

[Julian]

I think cross-draft looks right for that test yeah, at least at a first glance.

[gregsdennis]

(It should also be noted that this can still happen in our stable spec world if the embedded schema's meta-schema declares a different vocabulary.)

[gregsdennis]

@jdesrosiers @Julian, so cross-draft seems like a good place, assuming that the current test structure can support a meta-schema validation, but.... can the current test structure support a meta-schema validation (as opposed to just a schema validation of data that happens to also be a schema)?

[jdesrosiers]

can the current test structure support a meta-schema validation (as opposed to just a schema validation of data that happens to also be a schema)?

I don't think it needs to. The tests doesn't need to validate the schema directly, what matters is the result of using the schema to validate instances. The example given at the top is a good example. An implementation that doesn't switch dialects properly will error on that schema which means they fail the test as expected. An implementation that does switch dialects properly should be able to pass a test that expects { "foo": ["a", 42] } to validate to true.

If we switch the dialects on that example and put draft-07 at the root and 2020-12 embedded, we can't test that an implementation should consider that schema invalid, but that's not a new problem for the test suite. In general we can't test for schemas that we expect to be invalid.

[gregsdennis]

Okay. So we'll have to get creative, but the current setup can handle it.

(Interestingly, my implementation fails that test. Adding that to the list of things to do today...)

I wonder if these need to be optional tests because an implementation may not support (e.g.) draft 7.

[jdesrosiers]

I wonder if these need to be optional tests because an implementation may not support (e.g.) draft 7.

Yeah, that's the reason I haven't contributed tests like this a long time ago. It depends on what dialects an implementation supports. We could define a custom dialect, but that leaves out implementations that don't support that even tho that functionality isn't actually a dependency of this behavior.

[Julian]

cross-draft.json is already in the optional directory (precisely for the above reason, that it depends which drafts an implementation supports).

[gregsdennis]

I think we may need to update Core 8.1, which contains

A schema MUST successfully validate against its meta-schema

Maybe it just needs a link to 9.3.3 (add "as per section 9.3.3" or something)?