proposal: encoding/json: reject unknown fields in Decoder

[okdave]

Currently, json.Unmarshal (and json.Decoder.Decode) ignore fields in the incoming JSON which are absent in a target struct.

For example, the JSON { "A": true, "B": 123 } will successfully unmarshal into struct { A bool }.

This makes it difficult to do strict JSON parsing. One place this is an issue is in API creation, since you want to reject unknown fields on incoming JSON requests to allow safe introduction of new fields in the future (I've had this use case previously when working on a JSON REST API). It also can lead to typos leading to bugs in production: when a field is almost always the zero value in incoming JSON, you might not realise you're not even reading a misspelled field.

I propose that a new method is added to Decoder to turn on this stricter parsing, in much the same way UseNumber is used today. When in strict parsing mode, a key in the incoming JSON which cannot be applied to a struct will result in an MissingFieldError error. Like UnmarshalTypeError, decoding would continue for the remaining incoming data.

d := json.NewDecoder(r.Body)
d.UseStrictFields()
err := d.Decode(myStruct)

[bradfitz]

SGTM. @rsc, @adg?

[extemporalgenome]

Could we add a mechanism to the json package that allows a struct to declare that it should have strict or non-strict handling, independent of how the decoder is configured? This could be an interface, or a pair of zero-size types (or a bool type) that could be embedded -- in much the same way that the xml package provides configurability -- or perhaps a json struct tag that could be recognized on any padding field.

There are cases where it's desirable for some json objects in a "document" to be treated strictly, whereas others in the same document should be non-strict (yet still warrant use of a struct instead of a map). Allowing an "overflow" map field would make it easier to express this (if you include such a map field, and turn on strict handling, the containing struct would be non-strict).

[rsc]

This is a duplicate of #14750. Continuing @extemporalgenome's thought, I wonder if the strictness should always be in the tag instead of a new method. Per-struct is difficult but per-field is fine. You could imagine json:"Foo,exactname".

[bradfitz]

@rsc, partial duplicate. This bug is about undeclared fields. #14750 is about differing case.

[rsc]

Ah, thanks very much. OK, then UseStrictFields is definitely a bad name. :-)

We could certainly solve Dave's problem but I wonder if we should do something a bit more generic. In the XML decoder there is a ",any" tag that you can use to say "unexpected things go here". I wonder if JSON should allow a map[K]V (K=string, V=interface{} would be common but not strictly required) to have a tag json:",any" to collect otherwise ignored key-value pairs. Then the decoder can do whatever it needs to do with them. Similarly on encoding those would have to be reinserted into the marshalled object.

It might be too late for Go 1.7 to work all this out.

[cespare]

The "overflow" map discussed by @extemporalgenome and @rsc has been requested a few times. In fact, #6213 already exists for such a proposal. (The confusing name "inline" for the feature makes it hard to discover.) Perhaps this bug should supersede #6213.

[cespare]

I like the overflow map as a solution to the extra keys problem.

I think the dual of this issue is also important: sometimes you want to know that a key was missing from the JSON object. (In the case of a misspelling as in @okdave's motivation, you could detect either the missing key or the extra key.)

Adding a way to error on missing JSON keys is tracked by #6901. I believe these should be considered at the same time.

[okdave]

An overflow map would work for this purpose. One downside to adding that as a struct field would be that for deeply nested structures, you'd need to add the field to multiple structs and do a lot of manual bookkeeping after unmarshalling. One big advantage is no (terribly named) new method on Decoder and that it works with Unmarshal too.

Would that same map be used to add additional keys during Marshal? Or ignored there?

[extemporalgenome]

@okdave users would likely expect any Marshal(Unmarshal()) to produce output equivalent to the input.

[okdave]

Yup, I agree. @rsc I wasn't aiming for 1.7, but I'm happy to jump on this if there's consensus on this approach.