Binary versions in the Scala 3.x era

[sjrd]

Until Dotty 0.x.y-RC1's, the binary and TASTy formats were free to be broken at any time, and hence the binary version was 0.x (for example 0.27). With 3.0.0-M1, it has become 3.0.0-M1, which corresponds to what the milestones and RCs of Scala 2 do (a full version number).

We need to decide what will happen in Scala 3.0.0 and onwards. There are a few things to take into account.

• There exists a version 3.T.0 at which we will break and finalize TASTy, as well as change the stdlib to use Scala 3 features in appropriate places. That implies that it will also break binary compatibility.
• For all versions >= 3.0.0 and < 3.T.0, we will preserve binary compatibility, which also implies staying binary compatible with Scala 2.13.
• For all versions >= 3.T.0, we will keep backward tasty compatibility, basically forever. We don't know exactly what promises will be in terms of binary compat, but most likely we'll want to at least keep backward and forward binary compat between patch versions.

Therefore, we have at least two binary versions to settle on: the one for < 3.T.0, and the one for >= 3.T.0. I see the following options:

• 3.0 and 3.T. That choice means: the first minor version that uses the new binary version. It has the benefit that it can develop further in time if we decide that binary versions should stick to binary compatibility, not to TASTy compatibility, even after 3.T.0, since we can introduce another 3.U at some point. It is however confusing that 3.1 is compatible with binary version 3.0 but not 3.T.
• 3.0 and 3: That choice bets on 3 being the ultimate binary version, which will never change again. The use of 3.0 is a transient state in the meantime. It will be weird that 3.0.0 cannot read 3, but that's not too bad.
• 3 and 4: After all, there is no hard requirement for the binary version to be related to the general version. That choice makes it explicit that a binary version is not structured. It's flat: when it changes, the whole ecosystem has to be rebuilt, no matter what. (If we preserved backward compatibility, we wouldn't change the binary version to begin with.)
• And finally, 2.13 and 3. This one will make people scream, but it makes perfect sense given that Scala 3 and 2.13 are forward and backward binary compatible. The ecosystems are therefore not split: it is the same ecosystem. That also makes withDottyCompat and the reverse one for which we don't have a name yet both moot. It does mean that a library cannot release the same version separately for 2.13 and 3. But after all, it shouldn't have to, since we made everything so that the binary built by one version is usable from the other version.

[dwijnand]

I like 2.13 and 3, because it's ambitious. However it makes it impossible for a library to be cross-built for 2.13 and 3, so a library must decide if it's publishing for one or for the other...

Riffing off your ideas, I kind of like "3 and 3.T" but where "T" is literally "the Scala 3 Tasty" binary version (not, for example "3.6"). It's also mirrors OnePlus's phone names (3T, 5T, etc).

[odersky]

One possible problem might be that we want to allow backward compatibility between minor versions. So if 3.0 is binary 2.13, what is 3.1? It can't be 2.14 because that would imply a global break according to the Scala-2 scheme and it can't be 2.13.x either since that would imply backward and forward compatibility. Do I understand that correctly?

[sjrd]

One possible problem might be that we want to allow backward compatibility between minor versions. So if 3.0 is binary 2.13, what is 3.1? It can't be 2.14 because that would imply a global break according to the Scala-2 scheme and it can't be 2.13.x either since that would imply backward and forward compatibility. Do I understand that correctly?

It would be 2.13. You only change the binary version when you break backward binary compatibility, not when you only break forward compatibility. The reason for that is that when you keep backward binary compatibility, libraries do not need to be republished, so it is the same ecosystem.

This scheme has worked without any hiccup for Scala.js 0.6.x and 1.x: we've broken forward binary compatibility many times during the 0.6.x cycle, and already 3 times during the 1.x cycle. But 0.6.x has always used the binary version _sjs0.6 and 1.x has always used _sjs1.

[smarter]

I think using 2.13 as our binary version is not worth it: library maintainers do want the freedom to publish versions tailored to each compiler (which might require using different sources), forcing everyone to choose one means potentially compromising the user experience on the other one, see for example what @djspiewak wrote in https://gist.github.com/djspiewak/b8f2b4547951442102488964bc351cf9

I think all the other suggestions are fine, but I find using 3.0 for things which are not 3.0 is confusing, so I would rather go with 3abi1 which we would eventually bump to 3abi2 (or 4 if we decide to go in that direction), this makes it clear that the suffix is about binary compatibility and not source or tasty compatibility (as 3.1 would probably bump the minor tasty version).

[odersky]

It would be 2.13. You only change the binary version when you break backward binary compatibility, not when you only break forward compatibility. The reason for that is that when you keep backward binary compatibility, libraries do not need to be republished, so it is the same ecosystem.

But I have thought that for Scala 2.x, it's a stricter rule that also requires forward compatibility?

[sjrd]

I think using 2.13 as our binary version is not worth it: library maintainers do want the freedom to publish versions tailored to each compiler (which might require using different sources), forcing everyone to choose one means potentially compromising the user experience on the other one, see for example what @djspiewak wrote in https://gist.github.com/djspiewak/b8f2b4547951442102488964bc351cf9

Maintainers might want that freedom, but if they actually use it for anything non-trivial (i.e., where the binary API is not 100% the same), they might prevent the downward ecosystem from actually leveraging the bidirectional interop we built between Scala 2.13 and 3. Because an application A might use libraries X (2.13) and Y (3), both depending on the upstream library L which has variants for 2.13 and 3. And then A ends up with a non-reconcilable classpath.

[sjrd]

It would be 2.13. You only change the binary version when you break backward binary compatibility, not when you only break forward compatibility. The reason for that is that when you keep backward binary compatibility, libraries do not need to be republished, so it is the same ecosystem.

But I have thought that for Scala 2.x, it's a stricter rule that also requires forward compatibility?

That is not an issue, because it only applies to a) the binary encoding produced by the compiler, which is anyway frozen and b) the binary API of scala-library.jar, which is frozen as well. It doesn't apply to other libraries, in particular not scala3-library.jar.

[djspiewak]

My two cents…

From a user's perspective, the question to be answered is, given their set scalaVersion value, is this specific artifactId binary compatible. Thus, given version 2.13.3 (for example), is cats-effect_2.13 a compatible version? Of course, the answer to this is yes, and we can determine this by the fact that the bincompat version suffix is a strict prefix of the actual Scala version within the build.

Note that we don't need to answer the inverse question: for a given bincompat suffix, enumerate all compatible Scala versions. That's an interesting question, but one that never, in practice, arises. This asymmetry suggests that the 3 scheme is the correct one, with only one caveat.

For all versions of Scala 3 prior to 3.T (where we finalize Tasty), the bincompat suffix should mirror the Scala 2 scheme of including major.minor. For example: cats-effect_3.0. Once Tasty is finalized for the entire 3.x future-lineage, the bincompat scheme should drop the minor component: cats-effect_3.

The caveat here is that build tools need to understand this cliff so as to properly warn users that 3 is not a bincompat version for scalaVersion = 3.0.0, despite the fact that it's a strict prefix. These cliffs sunset though as builds are upgraded.

Note that Scala actually went through this kind of transition once before. I can't remember exactly when the shift took place (I believe it was during the 2.8.x line), but way back in 2.7.x days the bincompat suffix included the full Scala version since compatibility was broken with absolutely every release. It wasn't until later that Scala's own compatibility guarantees were hardened such that libraries could publish once for a major.minor version. This same scenario (with build tools having to special-case a particular threshold) was dealt with and, in practice, didn't really seem to create any headaches.

What is going to happen with Tasty is quite analogous to this compatibility hardening nearly a decade ago, except the minor component is now being dropped.

---

Also to @smarter's point, I would discourage everyone from thinking of the 3.0 and 2.13 ecosystems as being merged. They really are not. There are several reasons why this is the case, but very briefly:

• Transitive diamond eviction is completely broken when you have dependencies which are cross-published to both. ScalaCheck is a major and current exemplar of this, since 1.15.1 is published for 3.0.0-M1 and 2.13, Discipline has done the same, but Specs2 is only published for 2.13 and thus depends on the 2.13 version of ScalaCheck. This in turn means that anything downstream of both Discipline and Specs2 is forced to exclude ScalaCheck 1.15.1 2.13. If you don't exclude, Sbt will produce an error in dependency resolution which is modestly semantic in nature, but it doesn't give you any help in resolving it.
• Any Scala 3 migration issues which would have been a roadblock when doing an actual build migration in a given library are forced onto all downstream users if that library choses to continue publishing for 2.13 and forces their users to employ withDottyCompat. This is kind of analogous to the difference between static and dynamic typing. withDottyCompat is a bit like a runtime cast, wherein your users will discover issues in your API which you would have otherwise discovered for yourself. This is a lot more common than you would think, particularly in any type signatures which involve skolems, since NSC is immensely buggy in these areas while Dotty is much less so. Put bluntly: a not-insignificant percentage of published Scala libraries are unknowingly exploiting bugs in NSC. That exploitation needs to be either addressed by the library authors (by publishing for Scala 3), or by all of their downstream users (using withDottyCompat). In other words, withDottyCompat is a very dangerous thing for a library author to suggest unless they've actually completed the Scala 3 migration in library code, at which point they may as well publish the artifacts.
• The 2.13 unpickler has been found to have some issues (which I need to file… mea culpa on that one). Given the differences between Dotty and NSC, it wouldn't surprise me if there are more of these left undiscovered. At the very least, it's an unnecessary ecosystem risk.

Tldr, withDottyCompat is useful, but doesn't work as well as well as you probably think it does. 3.0 is a distinct ecosystem from 2.13, and what the ecosystem is compelled to do at this point is essentially the same as any of the previous 2.(n - 1) -> 2.n upgrades.

[sjrd]

From a user's perspective, the question to be answered is, given their set scalaVersion value, is this specific artifactId binary compatible. Thus, given version 2.13.3 (for example), is cats-effect_2.13 a compatible version? Of course, the answer to this is yes, and we can determine this by the fact that the bincompat version suffix is a strict prefix of the actual Scala version within the build.

I disagree. The real question is: given a full version (e.g., 2.13.3), what is the binary suffix that I need to use to resolve %% dependencies (here _2.13). That implies that there is only one possible answer (not several compatible binary suffixes).

For all versions of Scala 3 prior to 3.T (where we finalize Tasty), the bincompat suffix should mirror the Scala 2 scheme of including major.minor. For example: cats-effect_3.0. Once Tasty is finalized for the entire 3.x future-lineage, the bincompat scheme should drop the minor component: cats-effect_3.

IIUC your comment, that would mean that Scala 3.1.0 would use binary suffix _3.1. That is not an option, because 3.1.0 would be backward binary compatible with 3.0.x, and therefore must have the same binary version as 3.0.0.

The caveat here is that build tools need to understand this cliff so as to properly warn users that 3 is not a bincompat version for scalaVersion = 3.0.0, despite the fact that it's a strict prefix. These cliffs sunset though as builds are upgraded.

There will be at least one cliff regardless of the version scheme we use. But I agree with you that we can teach build tools to handle the cliffs. That's not an issue we need to be concerned about.

• Transitive diamond eviction is completely broken when you have dependencies which are cross-published to both. ScalaCheck is a major and current exemplar of this, since 1.15.1 is published for 3.0.0-M1 and 2.13, Discipline has done the same, but Specs2 is only published for 2.13 and thus depends on the 2.13 version of ScalaCheck. This in turn means that anything downstream of both Discipline and Specs2 is forced to exclude ScalaCheck 1.15.1 2.13. If you don't exclude, Sbt will produce an error in dependency resolution which is modestly semantic in nature, but it doesn't give you any help in resolving it.

That would be solved by using 2.13/3. That is actually the main point of using 2.13/3.

• Any Scala 3 migration issues which would have been a roadblock when doing an actual build migration in a given library are forced onto all downstream users if that library choses to continue publishing for 2.13 and forces their users to employ withDottyCompat. This is kind of analogous to the difference between static and dynamic typing. withDottyCompat is a bit like a runtime cast, wherein your users will discover issues in your API which you would have otherwise discovered for yourself. This is a lot more common than you would think, particularly in any type signatures which involve skolems, since NSC is immensely buggy in these areas while Dotty is much less so. Put bluntly: a not-insignificant percentage of published Scala libraries are unknowingly exploiting bugs in NSC. That exploitation needs to be either addressed by the library authors (by publishing for Scala 3), or by all of their downstream users (using withDottyCompat). In other words, withDottyCompat is a very dangerous thing for a library author to suggest unless they've actually completed the Scala 3 migration in library code, at which point they may as well publish the artifacts.

I don't think so. Only roadblocks that surface in the public/protected API of the library are forced on downstream users. But I would certainly hope that those an overwhelming minority, compared to roadblocks that impact the implementation of the library (in particular inside methods). Exploiting bugs in nsc is often inside methods (pattern matching unsoundness for example), not so much in the APIs. And I may be wrong but I don't think it's even possible for a skolem to be part of a public API, is it?

You maintain more libraries with complicated stuff than I do, so maybe I'm wrong on all accounts, here.

• The 2.13 unpickler has been found to have some issues (which I need to file… mea culpa on that one). Given the differences between Dotty and NSC, it wouldn't surprise me if there are more of these left undiscovered. At the very least, it's an unnecessary ecosystem risk.

We should definitely fix issues.

---

I may come off as defensive here. We've worked hard on designs and implementations to make this two-way interoperability between Scala 2.13 and 3. We should capitalize on it unless there is strong evidence that it's broken.

[mpilquist]

Unless I'm misunderstanding, the 2.13 & 3 compatibility does not support macros, which then forces many libraries to publish for both 2.13 and 3. Current list of macro libraries: https://scalacenter.github.io/scala-3-migration-guide/docs/macros/macro-libraries.html

[sjrd]

A library that has macros can put both the Scala 2 and Scala 3 macros in the same artifact if it is compiled by Scala 3.