Vendor external dependencies for tests

[seagreen]

Right now the tests reference files on external servers, such as here (which references the schema for JSON schemas). Note that this isn't in refRemote.json where it might be more expected, instead it's in plain ref.json.

This works OK, but isn't great for users with choppy internet connections because their tests become non-deterministic. Might it make more sense to vendor the files that are currently being referenced remotely, adding them to the ./remotes/ directory so they can be served up at localhost?

[gavinwahl]

I need this too, because I'm implementing json schema in a language that can't implement remote refs, so I would like to skip those tests.

[epoberezkin]

Spec says that although schema ids are URIs it doesn't mean that the schema should be downloadable from this URI. For these tests to work a validator should provide an API to associate any given schema with any URI. Skipping the tests (as well as executing them) is out of scope of this package.

I suggest to close the issue.

[handrews]

@epoberezkin I think that's what @seagreen is proposing- keep them locally and have the test framework load them from files rather than use an HTTP request. That seems pretty reasonable to me.

[seagreen]

@handrews: Thanks that was exactly what I meant. Just changed the title to be less confusing.

[epoberezkin]

@seagren @handrews I am not sure I follow. Tests we have here do not use Internet access. The fact we have references to external schemas in these tests does not mean that:

• they are loaded by the tests here
• they should be loaded by whatever test runner users use to iterate these tests

Given that the schemas that you refer to are meta-schemas that are present in json-schema-org/json-schema-spec I do not think it is a good idea to duplicate them in this repo. Instead whatever code is used to execute these tests should include them as dependencies from json-schema-spec

[seagreen]

@epoberezkin: That makes sense to me. On the other hand it really is a remote reference, and so wouldn't refRemote.json be a better fit for it?

[handrews]

@seagreen @epoberezkin whatever y'all decide is fine with me, I just jumped in because I thought i saw a misunderstanding. Carry on! :-)

[epoberezkin]

@seagreen if you are referring to the test here: https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/master/tests/draft4/ref.json#L177 I absolutely don't mind moving it to refRemote, although this split between "ref" and "refRemote" always seemed a bit unnecessary to me :)

But if we keep the split moving this test there makes sense

[seagreen]

@epoberezkin: Yep, that's the one!

[gavinwahl]

I see, so a validator is expected to provide an interface for specifying the external schemas? ie

validate_json_schema(
    data=data,
    schema=schema,
    external_refs={
        'http://json-schema.org/draft-04/schema#': {...},
    },
)

[seagreen]

@gavinwahl: I think the idea is that pre-mapping URIs to schemas is something validators have always been allowed to do, and that it's one way to perform this tests (with the other way being an HTTP request like normal).

[Julian]

Yeah that was part of the idea here. For reference, moving to a local schema would have caused my validator to make extra HTTP requests (or need extra test code), since the meta schemas are always cached.

And the difference between ref and refRemote was basically exactly this -- anything in refRemote is something you can expect to live in the remotes/ folder and requires the web server, and ref just lives inline in the same file.