Long-term compatibility plans for Scala 3

blog/_posts/2022-04-12-scala-3.1.2-released.md

@@ -9,6 +9,8 @@ We are happy to announce the release of Scala 3.1.2. This new release brings a f
 
 ## Configurable Scala output version
 
+**August 2022 Update:** configurable Scala output version turned out to be really hard to maintain and is not solving some of problems it was intended to solve. We have decided to drop it in 3.2 and propose long term support versions instead. You can read more in [the dedicated post]({{ site.baseurl }}/blog/2022/08/17/long-term-compatibility-plans.html).
+
 First and foremost, after months of brainstorming and testing, we are bringing an experimental possibility to configure the Scala output version. This means that the compiler can now generate TASTy files and classfiles in a format used by an older minor version of Scala (which can currently be only 3.0.x) effectively specifying the minimum version of the compiler that is able to read your compiled code.
 
 If you are a library maintainer, you may have faced a dilemma, should you update the Scala version to 3.1 or stay on 3.0. On the one hand, updating would bring you all bug fixes and enhancements from the newest version of the compiler. On the other, all new releases of your library would be unusable for all the users that are, for some reason, stuck on Scala 3.0.

blog/_posts/2022-08-17-long-term-compatibility-plans.md

@@ -0,0 +1,106 @@
+---
+layout: blog-detail
+post-type: blog
+by: Paweł Marks, VirtusLab
+title: Long-term compatibility plans for Scala 3
+---
+
+## Key takeaways
+
+- Scala 3 compiler that you are using to compile your code should be treated the same way as any library dependency in your project. You shouldn't be afraid of updating it. Bumping the patch version (e.g. 3.1.2 -> 3.1.3) is a no-brainer. Bumping the minor version (e.g. 3.1.3 -> 3.2.0) requires the same considerations as changing the minor version of any other dependency but is generally preferable.
+- You should not cross-compile between Scala 3 versions.
+- If you are maintaining a library, you should drop Scala 3.0. Also, you should upgrade to Scala 3.2 in your next minor release.
+- We are testing that the code passing the compilation with previous versions of the compiler will keep compiling in future releases, unless the fact that the code was compiling was a bug in the compiler. We are using hundreds of real-life projects to test that.
+- After evaluation, we have dropped the experimental `-scala-output-version` flag, as it turned out to be too hard to maintain and potentially harmful to the ecosystem.
+- Soon, we will start releasing Long-Term Support versions of the compiler for the users that require enhanced stability.
+
+## About compatibility
+
+Compilers, as all software really, are constantly evolving. Since the initial release of Scala 3, there have been seven stable versions of the compiler. Each has brought performance and stability enhancements, fixed bugs, improved reporting or introduced some new experimental features to the language. Nevertheless, we still treat all those versions as an implementation of the same language. What allows us to think in such a way is the compatibility guarantees.
+
+The term *compatibility* is often used quite vaguely. In the Scala world, the first intuition for this term could be phrased as follows: a new version of Scala is compatible with a previous version of Scala if a program that used to work with the earlier version still works with the latest version. Let us examine the term *compatibility* closer right now. For Scala 3 compiler, *compatibility* involves two concepts: source and output compatibility.
+
+We say that two compiler versions are source compatible if every single project that can be compiled with one version can also be compiled with the other, and the resulting programs behave the same way. We put a lot of effort into ensuring that source compatibility is preserved between patch releases, and that every infringement between minor versions is easy to fix. To enforce that, we are running builds of hundreds of open-source Scala 3 projects every night. Moreover, after every release candidate, we are testing the compiler on practically all open-source projects that were ever published for Scala 3.
+
+In addition, breaking source compatibility is usually not a big deal, as it can easily be detected at compile-time. In the past, fixing those problems have usually been limited to specifying explicit types in some places where the programmer relied on type inference. If your project is affected by a source-incompatible change, you often need to make only small adjustments. In the worst-case scenario, you will be locked on one version of the compiler for some time. That, however, will not affect projects depending on your code.
+
+Output compatibility (sometimes called *binary compatibility*, especially in Scala 2 contexts) is much more tricky and essential for our long-term plans for Scala. It can be understood as an ability of one version of the compiler to use the output (binaries and TASTy files) generated by the other version of the compiler and understand it correctly. To be more precise, we will further subdivide output compatibility to forward and backward. We say that the two versions are forward compatible when the older compiler can depend on the output of the newer one. Conversely, backward compatibility means that the newer compiler can use the output of the older version. In the rest of this post, the term *compatibility* will always refer to output compatibility.
+
+Scala 3 guarantees backward compatibility between all releases and forward compatibility between patch releases in the same minor line. That means that Scala `3.b.y` can consume the output of Scala `3.a.x` only if `b` is greater or equal to `a`. For example, the output of the Scala 3.0.2 compiler can be used by Scala 3.0.1 or 3.1.0. On the other hand, code compiled with Scala 3.1.0 cannot be a dependency of any project compiled with 3.0.2.
+
+## What version of Scala should I use?
+
+So, we know what output compatibility is and what Scala 3 guarantees. What does it mean in practice? What version of the compiler should I choose for my project? The compiler team believes that the good answer for this question is always *the newest one*.
+
+We believe that you should treat the compiler as one of your dependencies. If you are fine with updating the versions of your library dependencies, there should be no reason not to update the compiler version. In most cases, you wouldn't even notice that the Scala version has changed. Meanwhile, you can get better compilation performance, improved error messages, and most importantly, you may be able to use newly stabilized features of the language and of the standard library.
+
+If you are a library author following the semantic versioning, we advise you to introduce new minor versions of the compiler only in minor releases of your library. New patch versions of the compiler can be introduced anytime, as they are forward and backward compatible.
+
+That may be surprising if you have experience developing libraries for Scala 2. For example, migrating any codebase from 2.11 to 2.12 was a huge undertaking. Since they were not output-compatible, most libraries needed to resort to cross-compilation to support both versions. Thanks to the guarantees of Scala 3, there is no longer a need for cross-compilation. For example, the migration from Scala 3.1 to Scala 3.2 for the large majority of the projects would only require changing the version in the build definition. We have tested 900 Scala libraries that were passing compilation using 3.1.3. Out of them, only 13 failed to compile with 3.2.0-RC3. That means that for 98.6% of the libraries bumping the compiler version to 3.2.0 will not require a single change in the source code.
+
+## The problem
+
+While this sounds good, there is one problem with intertwined ecosystems. Even though we do what we can to make the transition between minor versions of the compiler as smooth as possible, there will always be projects stuck on older versions for various reasons. That means they may be unable to update some of their dependencies if said libraries updated themselves to the newer compiler. That, in order, encourages libraries to stay on the oldest possible version of the compiler, which bars them from potentially beneficial improvements in the newer compilers.
+
+For example, let us consider a project that is stuck on Scala 3.1, for some reason. We depend on version 1.3.5 of a library compiled with Scala 3.1. The library publishes version 1.4.0 with the compiler updated to 3.2. We cannot update our dependency because of output incompatibility. That also means that all patches for security issues won't be available unless the library author decides to backport them and release them as version 1.3.6. With that knowledge, the library author would be very reluctant to migrate from Scala 3.1 to 3.2.
+
+## Searching for a solution
+
+We have been discussing and testing various potential solutions in the past few months. Today we want to tell you more about the attempts that failed and present the solution that we believe is the best for the future of Scala.
+The three main criteria we were using to examine potential solutions were:
+
+- The burden on library authors - we don't want library maintainers and authors to feel overwhelmed by making frequent changes to the compiler that require actions on their side
+- The burden on the compiler maintainers - the compiler team has limited resources, so if maintaining compatibility required a considerable amount of work, we would have much less time to spend on improvements and bug fixes in the compiler
+- Innovativeness - some features require updating the minor version of the compiler (native image-friendly lazy vals being the latest example), and we want them delivered to the users as soon as possible. Also, we want important bug fixes to be delivered to users regardless of their update policy.
+
+## What we have tried so far
+
+### Full compatibility
+
+For a short time, we have considered releasing a patch version for each supported past minor line every time a new minor release of the compiler appears. That would allow every minor version of the Scala 3 compiler to accept the output of any other minor version (assuming the latest patch is used). This is something similar to what is happening in Scala 2 in terms of Scala 3 compatibility. However, applying this to all minor lines of Scala 3 would be an enormous amount of work, way beyond our capabilities.
+
+### Maintaining every minor release
+
+The lighter version of the approach mentioned above was to keep our current compatibility guarantees while providing backports of important bug fixes to the previous minor versions. That would still be a massive workload for us, and it would have the additional drawback of discouraging people from updating to the new minor releases.
+
+### Configurable output version
+
+This was the solution that we invested the most into. It was discussed in detail in [the previous blog post]({{ site.baseurl }}/blog/2022/04/12/scala-3.1.2-released.html#configurable-scala-output-version). To summarize: in 3.1.2, we have shipped a new experimental compiler flag that allowed developers to make the compiler generate output that is usable by older versions of the compiler. While this would allow all projects to receive important fixes, it would probably result in a strange state where all the libraries are released with output version 3.1, similar to how nearly all java libraries are still released with output version 8, even though Java 8 is eight years old.
+
+More importantly, during work on 3.2, we realized that maintaining this flag may be more challenging than anticipated. With time, it will be increasingly harder to be sure that our handling of it is correct. **Thus, we have decided to remove the possibility of configuring the output version altogether in Scala 3.2.**
+
+## Our solution: Scala LTS & Scala Next
+
+Finally, we have agreed that the best course of action would be to split Scala development into two lines, called *Scala LTS* and *Scala Next*. Both would live as separate branches in the compiler repository. Both will have separate but possibly synchronized releases. Porting changes from one to the other should be easy and common.
+
+Scala Next will be the line on which the language development is taking place. It will receive frequent minor updates, and all experimental language features will live there.
+
+Scala LTS, on the other hand, will be the stable long-term support line. An LTS version will only receive patch updates, which means that all releases of the same LTS line will be forward and backward compatible in terms of output. The only changes between releases will be bug fixes, non-language changes (doctool, semanticDB, reporting), and minor quality-of-life enhancements (only if we are sure we are not breaking any compatibility guarantees). After more than two years, we may decide to nominate one of the releases from the Next line as the new LTS. After that, we guarantee that the compiler team will support the previous LTS for at least one additional year. That means the support time for each LTS release is at least three years.
+
+We think this will allow projects to retain a more conservative approach to updating the compiler versions and still receive the bug fixes and some improvements.
+
+![Scala LTS Roadmap]({{ site.baseurl }}/resources/img/scala3-lts-roadmap.svg)
+
+### The compatibility guarantees in the new model
+
+The new model doesn't change our compatibility guarantees. We keep forward and backward compatibility within a single minor version and backward compatibility across different minor versions. That means that the code compiled with Scala LTS can be used as a dependency by all Scala Next versions newer than said LTS, but not the other way around.
+
+## The (future) best practices
+
+Once the LTS & Next model is in place, we suggest the following practices for different groups of users:
+
+### Owners of commercial projects
+
+We suggest you stick to the LTS version. It will give you the best tooling support and a stable compiler. If you are migrating a bigger project from Scala 2, you should migrate it straight to the newest LTS.
+
+You may also choose to be up to date with the latest minor version. That will allow you to use new features as soon as they are added to the language. However, you will need to be ready for frequent compiler updates. Otherwise, your project may end up with an unsupported language version.
+
+### Library maintainers
+
+Unless your library is built around some new language feature available only on Next, you should stick to the latest LTS version to have the broadest possible user base. If your library contains multiple modules, some of them may require Scala Next. It is perfectly fine as long as you keep in mind that modules compiled with Scala LTS cannot depend on them (the other way around is fine). Also, users using your library from Scala LTS won't be able to access modules built with Next.
+
+Remember that when you bump the minor version of the compiler (e.g., moving from the previous Scala LTS line to the next), you also need to bump the minor version of your library.
+
+### Authors of hobby or standalone research projects
+
+Feel free to use Scala Next. Then you will be able to use all libraries, both those for Scala LTS and Scala Next. You also will be able to test the newest and experimental features of the language.