TASTy format versioning scheme

[anatoliykmetyuk]

Currently, we are only using major and minor version numbers to identify the TASTy format version. We need to rethink this scheme before the release.

After the release, in versions 3.0.x, we will maintain both backward and forward compatibility of the format. However, there will be a transition period between 3.0.x and 3.1.x. During this period, there will be commits that break forward compatibility of the format in preparation for 3.1.x while still on 3.0.x. For this situation, we need to modify the TASTy versioning scheme to specify whether the version is the transition version. In general, we need to think of the logistics of TASTy evolution between 3.0.x and 3.1.x.

This issue is the place for this discussion. We need to come to a decision about it before the 3.0.0-RC1 release since we'd like to try out the new versioning format of TASTy in that release.

/cc @odersky @sjrd @smarter @bishabosha

[bishabosha]

@sjrd with the way Tasty unpickling is currently implemented it seems that nothing needs to be done to support the above proposed process:

• we keep same major version between two backwards compatible Tasty releases
• we increase the minor version in Tasty for forward incompatible (but still backwards compatible) releases, and so the compiler will reject Tasty with a higher minor version than it knows, but continues to support Tasty with a smaller minor version and same major version.

[sjrd]

Yes, that is true. But the limitation of the current scheme is that, when we read an invalid TASTy version, all we can tell the user is "version 7.45 is invalid, expected 7.40 or an earlier 7.x" or something like that. But that doesn't help the user know what version of Scala does 7.45 correspond to.

The idea here would be to align the TASTy versions better with the language version. So we could say "version 3.1 is invalid, expected 3.0; please upgrade to Scala to 3.1.0 or later".

But with that scheme comes the issue of "transient" TASTy versions: the ones in between 3.0.n and 3.1.0, when we're in snapshots/nighly builds and we've already broken forward compat, but we will do it again before 3.1.0. In that case we cannot use 3.0 nor 3.1. We would need something else that is "in the middle", and which is backward compatible with 3.0 but not compatible in any way with 3.1.

In Scala.js, we use 3.1-SNAPSHOT for that purpose, or 3.1-M1, etc. That requires a third component which is a string, though.

[smarter]

That requires a third component which is a string, though.

Since that's a bit complicated in a binary format, my suggestion is to just use a number for the third field, this field would be 0 for final versions and non-zero for preview versions.

[bishabosha]

It should be fine to put after the uuid (or minor version) the string length as a nat, then simply treat the next n bytes as UTF-8

[smarter]

Yes, but it's some extra complexity that might not be worth it.

[bishabosha]

I propose that an additional two Longs worth of header can be used to fully encode the same information as is currently displayed to the user when displaying the compiler version using -version flag, e.g. 3.0.0-M4-NIGHTLY-git-c2f24d2ee7938e59, we could call this a vendor string payload.

but then this does not account for some tool different to the scala compiler producing a tasty file

[bishabosha]

is the purpose of this extra field to simply display a better error message?, which would imply it needs to be able to decode a future version string that represents a scala compiler version for them to upgrade to.

[sjrd]

Yes, I don't think there's any other reason.

[bishabosha]

At the Dotty meeting we discussed that it would be desirable for the "data format" version to still be independent of the source language/library version.

• but it is still important to track the addition of primitive types in the compiler in a forward incompatible tasty version.

A point has also been made that we do not want to bump the minor version in a snapshot/nightly as that version is not stable/for-release, so we shouldn't be growing the minor version out of control.

So a potential plan - one extra field encodes that a tasty version is experimental, and observing that flag should cause a different error mode to if the version is not experimental.

There can then potentially still be an extra field for encoding a string of the compiler version, but the format has not been discussed.