Skip to content

Commit ea56862

Browse files
tgodziktgodzik
committed
docs: Add post mortem docs
1 parent 8fa013a commit ea56862

File tree

1 file changed

+140
-0
lines changed

1 file changed

+140
-0
lines changed

‎_posts/2024-10-24-post-mortem-3.6.0.md

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
---
2+
category: announcement
3+
permalink: /news/post-mortem-3.6.0.html
4+
title: "Postmortem of Scala 3.6.0"
5+
by: Scala Core Team
6+
---
7+
8+
Incident Date: _October 18th, 2024_
9+
Nature of the incident: _Accidental release of 3.6.0-RC1 under 3.6.0 on Maven
10+
Central_
11+
12+
On Friday, 18th of October 2024, during the publication process of **3.6.0-RC1**
13+
an incident occurred that ended up in publishing Scala **3.6.0** instead. After
14+
an internal investigation, we pinpointed the incident to be of technical nature.
15+
16+
Within one hour of catching the issue, the
17+
[Scala Core Team](https://scala-lang.org/scala-core/) mitigated the problem and
18+
started the Scala **3.6.1** release – that is to be treated as RC, which is
19+
available since Saturday 19th October. Scala **3.6.0** was abandoned and
20+
promptly announced to the public as broken.
21+
22+
We apologize to the Scala users for any inconvenience it might have caused. We
23+
are working on better understanding what led up to this accident and ensuring it
24+
doesn’t happen again.
25+
26+
We invite you to follow the recommendations below and find out more details
27+
about our findings of what happened and what is done to prevent future accidents
28+
of this kind.
29+
30+
## Recommendations
31+
32+
Use version 3.5.2, the current stable Next version, recommended for general
33+
use.
34+
Use version 3.6.1 as an RC version, to be used only for testing.
35+
DO NOT USE 3.6.0, not for testing, not for anything, it’s considered broken.
36+
37+
We are working on 3.6.2 as the new stable release, which will be available
38+
sometime in the second half of November.
39+
40+
## How does it affect users?
41+
42+
Scala 3.6.0 is a broken release \- it should never be used by users of Scala.
43+
The results of the compilation might not be consumed by stable versions of the
44+
compiler or TASTy based tooling.
45+
Tooling that automatically uses the latest available version of Scala will use
46+
the hot-fix release 3.6.1, which should be treated as a Release Candidate
47+
version. We don’t encourage most users to upgrade yet, unless for testing
48+
purposes. Instead, use the stable 3.5.2 and wait with upgrades until 3.6.2 is
49+
out.
50+
51+
## What happened?
52+
53+
During the publication of Scala **3.6.0-RC1** to the Sonatype Maven
54+
repository**,** a mistake occurred and we released **a misconfigured 3.6.0**
55+
instead. As a result, the released version of the compiler produces TASTy files
56+
in the experimental version that cannot be consumed by stable TASTy consumers
57+
(e.g. the Scala compiler, tasty-query, Scaladoc). Setting the TASTy version to
58+
an experimental one is a standard procedure when releasing a new release
59+
candidate of a minor version of Scala 3\.
60+
61+
## What immediate steps were taken?
62+
63+
On the evening of Friday the 18th of October 2024 the internal compiler team was
64+
informed of the erroneous release and the following immediate steps were taken:
65+
66+
- We announced on social media that Scala 3.6.0 was released unintentionally and
67+
is discouraged from being used.
68+
- We prevented `scala-steward` from automatically upgrading repositories using
69+
Scala Next to the 3.6.x line.
70+
- We followed up with a hotfix release in the form of Scala 3.6.1 on the
71+
following day.
72+
73+
## Why did we need 3.6.1?
74+
75+
Some tools in the Scala ecosystem can refer to the latest stable Scala version
76+
that’s been published on Maven Central. Those include:
77+
78+
- Scala CLI / the `scala` runner (via `-S 3` from the command line or with the
79+
`//> using scala 3` using directive)
80+
- coursier (with `cs setup` and `cs install scala` sub-commands)
81+
- potentially other tooling integrating with `coursier`
82+
83+
We were concerned Scala users might use this unknowingly broken version of the
84+
compiler to produce artifacts with the experimental version of TASTy. In
85+
particular, we wanted to prevent publishing Scala libraries from being published
86+
to the (immutable) Maven repository using a broken Scala release. It could also
87+
potentially affect applications using Scala 3.6.0 that would try to use
88+
libraries produced by Scala 3.6.1 or later.
89+
90+
We decided to publish Scala 3.6.1 as a follow-up ASAP so that the most recent
91+
version doesn’t include the experimental TASTy flag nor RC settings. With 3.6.1
92+
we ensured tooling will, at the very least, refer to a Scala version that
93+
actually could be considered stable.
94+
95+
We don’t yet encourage users to upgrade to Scala 3.6.1 on their own, except for
96+
testing purposes. Users should wait with upgrades until a future announcement.
97+
For now we advise users to stay on Scala 3.5.2 until further notice.
98+
99+
## Why did it happen?
100+
101+
The accidental release under the incorrect version happened due to a combination
102+
of multiple factors:
103+
104+
- Scala 3 uses an `sbt` plugin to automatically release artifacts to Sonatype
105+
Maven repositories. The automation happens without manual confirmation of the
106+
release. This factor prevented us from auditing published artifacts and
107+
removing the artifacts from the staging repository. Removing artifacts from
108+
Maven Central after they’ve been published is also essentially impossible.
109+
- We introduced changes to the algorithm for calculating versions of the
110+
compiler to mitigate problems with creating `.msi` installer files. As the
111+
change did not alter SNAPSHOT/NIGHTLY behavior, we did not detect the issue in
112+
time. We now realize similar changes require extra testing procedures, which
113+
will be implemented as an improvement. The change was added in
114+
[scala3\#21011](https://github.com/scala/scala3/pull/21011).
115+
- The change introducing the issue was merged on the 11th of July. Now, three
116+
months later, at the time of the release the side effects were not properly
117+
considered as in the meantime there was a change in staff coupled with
118+
insufficient documentation.
119+
- We experienced unrelated downtime on the side of our infrastructure, which
120+
delayed the (as it was meant to be) release of 3.6.0-RC1 from Wednesday 16th
121+
to Friday 18th of October. Given that it was an RC, we continued the queued
122+
job on Friday. We now realize we should have postponed it further until Monday
123+
21st. Resolving the incident would have been a lot easier for the Core and
124+
Compiler teams on a weekday rather than on a Friday night.
125+
126+
## How do we stop it from repeating?
127+
128+
In conclusion, after all that we learned from this incident, we are introducing
129+
the following processes as preventative measures:
130+
131+
- Extra checks to the release procedure to ensure the correct versioning of
132+
published Scala artifacts.
133+
- Any changes related to either release scripts or compiler versioning will be
134+
tested using a custom Maven repository before the start of a public release.
135+
- Additional, manual approval of published artifacts to the staging Maven
136+
repository.
137+
- We will be mindful of the day of week for releases, and avoid Thursdays and
138+
Fridays at all costs.
139+
- We will ensure that every change associated with the release process is well
140+
documented internally.

0 commit comments

Comments
 (0)