Should the spec suggest or recommend that validators only process json that validates against the metaschema?

[Relequestual]

As a result of a discussion at json-schema-org/JSON-Schema-Test-Suite#128 (comment), I ask if the spec should state that it is recomended that validation library confirms that the schema is valid against the metaschema before processing.
This would avlid issues like version missmatch and elements which may abigious, and the library has taken a speific stance on.

[iainbeeston]

My opinion is that the test suite should only include schemas that are valid, according to the draft metaschema. However, if the metaschema allows something (even if it's contradictory, like an {"type": "object"; "multipleOf": 5}) then there should be a test for it. I'd argue that we can already user validate schemas, by validating a user schema against the metaschema. So there is no need to handle invalid schemas in the test suite itself.

It would be nice if we could specify dependencies that depend on the value of a property (I don't think this is possible?). That would allow us to specify that if the value of the "type" property is "integer" then additonal properties are allowed, such as "multipleOf"

[awwright]

Normatively, what constitutes a valid schema and what is invalid is defined by the specification, not anything else.

Ideally, though, the meta-schema we supply will always be valid against a valid JSON Schema.

Perhaps the test suite should include cases that test that invalid schemas raise an error condition. There's not too many of these, though.

[Relequestual]

I think the simplest example was something like minimum. Although it's obvious that this applies only to numerical values, the metaschema doesn't specify the requirement. I think I've read somewhere that if the property doesn't apply, then it should just be ignored. My suggestion was that the metaschema COULD be more strict to, pick up on these types of invalid constructions.

I dislike the "it's wrong so ignore it" principal, as I feel it leads to unexpected behaviour, where a use believes the schema should do one thing, but it doesn't, and they don't know why.

[awwright]

I'm not sure what you mean, can you provide an example?

"minimum" is never ignored (so to speak), it just doesn't return invalid for instances that aren't numbers (i.e. it returns valid for things when the instance is something besides a number). The schema is perfectly valid.

If you mean that the value for the "minimum" keyword must be a number, that's already a MUST level requirement, and is also reflected in the meta-schema.

[iainbeeston]

I'd assume, based on the validation spec, that if you use a property that has no meaning for the given type, then it should be ignored, and we should have tests for that.

For example, section 5.1 is entitled "Validation keywords for numeric instances (number and integer)". All of the rules under that section assume that the value will be a number, and unfortunately it doesn't say what should happen if it is not a number.

[awwright]

@iainbeeston In the absence of an explicit invalid result or error, an instance is valid.

Earlier in the document, it says

When the primitive type of the instance cannot be validated by a given keyword, validation for this keyword and instance SHOULD succeed.

This has all been rewritten for the next draft, with the same behavior.

[Relequestual]

Right, that's what I'm talking about. I'm suggesting that it SHOULD NOT succeed, and an error is thrown that validation is trying to be applied to the wrong object type. If people dissagree with that, that's fine. It's just my thoughts on making the spec more strict and allowing people to pick up when they are doing something not quite right.

For example, having minimum on a string, I feel, should trigger an error. The metaschema chould formalise the relationship between properties and the types they apply to, by requiering that minimum is only valid with a numerical type. If that was defined in the metaschema, any library which validates a schema against the metaschema would trigger a validation error.

[awwright]

@Relequestual Errors are only raised if there's something wrong with the schema itself.

A valid schema always produces a valid or invalid result against an instance; an invalid schema always produces an error against an instance, with an indeterminate result.

The reason that non-numbers are ignored for things like "minimum" is so you can have schemas that validate against multiple types of instances:

{ minimum:0, minLength:1, type:["string","number"] }

[Relequestual]

OK. Hadn't considered that. In which case, that's a valid use case for saying no to my specific question regarding ignoring validation properties which are type specific.

[iainbeeston]

So, can I clarify what the outcome of this should be? Can we add more tests to define what should happen when a property (eg. multipleOf or minimum) is used for the wrong type?

[awwright]

Most of the tests should already verify the correct behavior, e.g.:

• https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/5fb3d9f1a1c4136f544fbd0029942ea559732f8e/tests/draft4/additionalItems.json#L65
• https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/5fb3d9f1a1c4136f544fbd0029942ea559732f8e/tests/draft4/additionalProperties.json#L22
• https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/5fb3d9f1a1c4136f544fbd0029942ea559732f8e/tests/draft4/dependencies.json#L29
• https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/5fb3d9f1a1c4136f544fbd0029942ea559732f8e/tests/draft4/items.json#L19
• https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/5fb3d9f1a1c4136f544fbd0029942ea559732f8e/tests/draft4/maxLength.json#L22
• https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/5fb3d9f1a1c4136f544fbd0029942ea559732f8e/tests/draft4/minimum.json#L17

If we're missing one, then yes, let's add a test case.

[Relequestual]

@iainbeeston the correct behaviour is to ignore said property, as it's possible to have an array of possible types. This is as intended.

I guess it could be possible to write a metaschema such that if it's an array of types, that one of them must be a numerical type for property minimum to be allowed, for example, but it DOES create more complexity.

Given that there's a valid use case (array of possible types), I'm satisfied that it's not just an oversight, and as such, will close this issue.

[awwright]

That's probably the biggest use case, but the real reason is actually because that (1) verifying the type is of the expected type, and (2) verifying (for example) a string has a minimum length, are two different tests that one might want to compare.

Or to put it another way, why should "minimum" invalidate all non-number instances, when the "type" keyword is perfectly capable of doing that instead?

[Relequestual]

I was suggesting that the validator should fails the given schema based on the fact it doesn't validate against the metaschema. I was suggesting the validator shouldn't even start parsing the JSON till the Schema is considered valid.

Actually, that was what this issue was about, not the specific example. The spec should recommend that validation of the schema against the metaschema should happen before any attempt to validate the json against the schema.