Standardized schema output (validation result, annotations, and errors) (cont.)

[gregsdennis]

This issue is to summarize the discussions and progress of #396, which sought to develop a standard output format for JSON Schema.

This issue does not cover standardized error message wording. This issue is for output formatting only.

---

Data Requirements

Validation outcome

This will be a simple boolean value indicating whether the instance passed validation.

Annotations/Errors

Since annotations are only collected when validation passes and errors only occur when validation fails, these two sets are mutually exclusive and will never appear in the result set together. However, they do share a similar (if not the same) structure.

• Keyword relative location - The location of the keyword that produced the annotation or error. The purpose of this data is to show the resolution path which resulted in the subschema that contains the keyword.
  • relative to the root of the principal schema; should include (inline) any $ref segments in the path
  • JSON pointer
• Keyword absolute location - The direct location to the keyword that produced the annotation or error. This is provided as a convenience to the user so that they don't have to resolve the keyword's subschema, which may not be trivial task. It is only provided if the relative location contains $refs (otherwise, the two locations will be the same).
  • absolute URI
  • may not have any association to the principal schema
• Instance location - The location in the JSON instance being validated.
  • relative to the root of the JSON instance
  • JSON pointer
• Value/Message - This is the annotation or error itself.

Instance data

We had discussed including the instance data in the response, but we determined that it was sufficient to provide a pointer (see Instance location above).

Copying the data into the output can still be an option for implementations, however.

Format

Both proposed formats return an object containing the validation outcome and a list of annotations or errors. The unresolved bit is how to represent annotations and errors generated from subschemas.

NOTE I will be using errors for these examples, but annotations could be used in their place. As mentioned before their structure is similar.

For these examples, we will use the following schema and (invalid) instances

// schema
{
  "$id":"http://schemarepo.org/schemas/user.json",
  "$schema":"http://json-schema.org/draft-07/schema#",
  "type":"object",
  "definitions":{
    "min18":{
      "type":"integer",
      "minimum":18
    },
    "username":{
      "type":"string",
      "minLength":8
    },
    "member":{
      "type":"object",
      "properties":{
        "age":{"$ref":"#/definitions/min18"},
        "username":{"$ref":"#/definitions/username"}
      }
    },
    "membershipTypes":{"enum":["admin","user"]}
  },
  "oneOf":[
    {
      "properties":{
        "member":{"$ref":"#/definitions/member"},
        "membershipType":{"$ref":"#/definitions/membershipTypes"}
      }
    },
    {
      "properties":{
        "membershipType":{"const":"guest"},
        "firstName":{"type":"string"},
        "lastName":{"type":"string"}
      },
      "additionalProperties":false
    }
  ]
}

// instance #1
{
  "member":{
      "age":5,  // doesn't meet minimum
      "username":"aName"  // doesn't meet minLength
  },
  "membershipType":"user"
}

// instance #2
{
  "membershipType":"guest",
  "firstName":"Charles",
  "lastName":"InCharge",
  "age":27  // additional properties not allowed
}

Flat

This proposal outputs all annotations and errors in a flat list. Because it's a flat list, it's easy to both build and consume.

// instance #1 results
{
  "valid":false,
  "errors":[
    {
      "keywordLocation":"#/oneOf",
      "instanceLocation":"/",
      "message":"the instance did not pass any of the subschema"
    },
    {
      "keywordLocation":"#/oneOf/0/properties/member/properties/age/$ref/minimum",
      "absoluteKeywordLocation":"http://schemarepo.org/schemas/user.json#/definitions/min18/minimum",
      "instanceLocation":"/member/age",
      "message":"value is too small"
    },
    {
      "keywordLocation":"#/oneOf/0/properties/member/properties/userName/$ref/minLength",
      "absoluteKeywordLocation":"http://schemarepo.org/schemas/user.json#/definitions/username/minLength",
      "instanceLocation":"/member/username",
      "message":"value is too short"
    },
    {
      "keywordLocation":"#/oneOf/1/membershipType",
      "instanceLocation":"/member/membershipType",
      "message":"value does not match the required value"
    },
    {
      "keywordLocation":"#/oneOf/1/additionalProperties",
      "instanceLocation":"/member/member",
      "message":"additional properties are not allowed"
    }
  ]
}

// instance #2 results
{
  "valid":false,
  "errors":[
    {
      "keywordLocation":"#/oneOf",
      "instanceLocation":"/",
      "message":"the instance did not pass any of the subschema"
    },
    {
      "keywordLocation":"#/oneOf/0/properties/membershipType/$ref/enum",
      "absoluteKeywordLocation":"http://schemarepo.org/schemas/user.json#/definitions/membershipTypes/enum",
      "instanceLocation":"/membershipType",
      "message":"value is not one of the required values"
    },
    {
      "keywordLocation":"#/oneOf/1/additionalProperties",
      "instanceLocation":"/member/age",
      "message":"additional properties are not allowed"
    }
  ]
}

Hierarchical

This proposal arranges the errors in a hierarchical format based on the schema.

The primary argument against the flat structure is that it's difficult (both for machines and humans) to see any association between the errors. For example, in instance #1 results above, there is no immediate indication that the first three errors pertain to the first subschema of the oneOf and the last two errors pertain to the second subschema. Moreover it becomes more difficult to understand that either the first three or the last two must be resolved to pass the instance, but not all five. The hierarchical format aims to make that association more apparent.

The construction of this object requires rules that would need to be included in the specification, particularly the conditions that are required for a node to be present.

1. All applicator keywords (*Of, $ref, if/then/else, etc.) require a node.
2. Any path branching requires a node. (Note that a branch node may not produce an annotation or error message).

An (unoptimized) algorithm for this may be

1. Build out the validation structure, creating nodes for all keywords in the schema.
2. As one travels back up the structure from the leaves, analyze the parent nodes as follows:
   1. If it is an applicator keyword, it must remain.
   2. If it contains multiple children, it must remain.
   3. If it contains no children, it is removed.
   4. If it contains a single child (only one descendant keyword produced an error), it is replaced by the child.

// instance #1 results
{
  "valid": false,
  "errors": [
    {
      "schemaLocation": "#/oneOf",
      "instanceLocation": "/",
      "message": "the instance did not pass any of the subschema",
      "errors": [
        {
          "schemaLocation": "#/oneOf/0/properties/member/properties",
          "instanceLocation": "/",
          "errors": [
            {
              "schemaLocation": "#/oneOf/0/properties/member/properties/age/$ref",
              "instanceLocation": "/member/age",
              "message": "referenced schema failed",
              "errors": [
                {
                  "schemaLocation": "http://schemarepo.org/schemas/user.json#/definitions/min18/minimum",
                  "instanceLocation": "/member/age",
                  "message": "value is too small"
                }
              ]
            },
            {
              "schemaLocation": "#/oneOf/0/properties/member/properties/userName/$ref",
              "instanceLocation": "/member/username",
              "message": "referenced schema failed",
              "errors": [
                {
                  "schemaLocation": "http://schemarepo.org/schemas/user.json#/definitions/username/minLength",
                  "instanceLocation": "/member/username",
                  "message": "value is too short"
                }
              ]
            }
          ]
        },
        {
          "schemaLocation": "#/oneOf/1",
          "errors": [
            {
              "schemaLocation": "#/oneOf/1/membershipType",
              "instanceLocation": "/member/membershipType",
              "message": "value does not match the required value"
            },
            {
              "schemaLocation": "#/oneOf/1/additionalProperties",
              "instanceLocation": "/member/member",
              "message": "additional properties are not allowed"
            }
          ]
        }
      ]
    }
  ]
}}

// instance #2 results
{
  "valid": false,
  "errors": [
    {
      "valid": false,
      "schemaLocation": "#/oneOf",
      "instanceLocation": "/",
      "message": "the instance did not pass any of the subschema",
      "errors": [
        {
          "valid": false,
          "errors": [
            {
              "schemaLocation": "#/oneOf/0/properties/membershipType/$ref",
              "instanceLocation": "/membershipType",
              "message": "referenced schema failed",
              "errors": [
                {
                  "schemaLocation": "http://schemarepo.org/schemas/user.json#/definitions/membershipTypes/enum",
                  "instanceLocation": "/membershipType",
                  "error": "value is not one of the required values"
                }
              ]
            }
          ]
        },
        {
          "valid": false,
          "errors": [
            {
              "schemaLocation": "#/oneOf/1/additionalProperties",
              "instanceLocation": "/member/age",
              "message": "additional properties are not allowed"
            }
          ]
        }
      ]
    }
  ]
}

Other considerations

Though the crux of this issue is the above, some other proposals have been made relating to this topic.

Implementation domain

The specification could allow for both of these formats, allowing the implementation to choose.

Configurable output levels

The specification could define different output settings.

1. Basic would be a response containing simply the boolean result indicating a pass/fail status.
2. List
3. Hierarchy
4. VerboseHierarchy would be an uncollapsed hierarchy (potentially useful for forms and other applications that would need a fixed path.)

[awwright]

Upon further consideration, I think it might be OK to have a standardized schema for reporting validation errors, especially for Web services, e.g. if you try to submit a document to a server, how the server would report back which problems should be corrected.

RDFa has a similar standard, it defines a vocabulary for the purpose of Web services to report errors while parsing RDFa-enabled documents.

[handrews]

Great point @awwright, and it might be interesting to think of how such a format might fit with RFC 7807 Problem Details for HTTP APIs (the application/problem+json media type).