New error metadata system required.

[michaelsproul]

It'd be nice to hang on to the error index in the Rust 1.2 release... currently it's disabled due to the flakiness of the metadata collation system (see #25706).

Amongst the issues with the previous implementation:

• Unsynchronised concurrent access of files, leading to the use of malformed JSON data.
• No way to determine the freshness of metadata files.
• Failure to write files (not thoroughly investigated). Probably related to stale files hanging around.

Before I go ahead and implement a crazy new system, I just want to check the sanity of my design:

• Alter the __build_diagnostic_array! invocations so that the identifier used for diagnostics arrays is something unlikely to be used elsewhere. This step isn't strictly necessary, as the current name DIAGNOSTICS is already quite unique.
• Register an internal lint (i.e. one that only runs against the compiler's source) that extracts error metadata from every item with the unique name used by __build_diagnostic_array!. Here we run into the same potential concurrency problem as before, because crates can be compiled in parallel. As long as the lint doesn't attempt to check uniqueness by reading files for other crates, the parallel writes to separate per-crate files should be ok. E.g. when running the lint against librustc_typeck, dump metadata to tmp/extended-errors/librustc_typeck.json. What worries me about this step is that it is essentially the same as dumping metadata from the __build_diagnostic_array! without uniqueness-checks, which we tried unsuccessfully (that's the unknown error).
• In the error-index-generator - which runs once all crates have been compiled - we can check uniqueness.

I still feel like this is very similar to the previous, failing system, and would like to alter it more drastically if anyone has any ideas...

cc @pnkfelix @alexcrichton @Manishearth

[alexcrichton]

As long as the lint doesn't attempt to check uniqueness by reading files for other crates, the parallel writes to separate per-crate files should be ok.

I do think that you can compile librustc_typeck in parallel with itself as well, however, for example with cross-compiled builds. I think that this worry could be alleviated by scoping the output to a target-specific location, however.

As long as the lint doesn't attempt to check uniqueness by reading files for other crates, the parallel writes to separate per-crate files should be ok.

I'd recommend putting the uniqueness part in the make tidy step personally, seems like a good scriptable task!

[michaelsproul]

I do think that you can compile librustc_typeck in parallel with itself as well, however, for example with cross-compiled builds. I think that this worry could be alleviated by scoping the output to a target-specific location, however.

Good idea!

I'd recommend putting the uniqueness part in the make tidy step personally, seems like a good scriptable task!

Yep, we'll just have to make sure it doesn't fail if the JSON files aren't there, like whenmake tidy runs before the compiler has been built. Our current infrastructure also doesn't seem to contain any rules to ensure that tidy is run last when running check, although it seems it happens this way because of the dependency ordering - check: ... tidy.

[steveklabnik]

These labels aren't perfect, exactly.

Anyway, yes, it would be really nice not to lose this. I've heard so many people describe this as an important thing to them.