Bisect script failures

[nicolasstucki]

For any Scala source, let us call it Test.scala the following fails

$ scala-cli project/scripts/bisect.scala -- compile Test.scala

Bisecting with validation script: /var/folders/j_/1jnflqdn6vg_6q2602rxq46m0000gn/T/scala-bisect-validator9193156609557343888
#####################################

#!/usr/bin/env bash
scala-cli compile -S "$1" --server=false Test.scala


#####################################

Verifying the first release: 3.0.1-RC1-bin-20210413-f3c1468-NIGHTLY
Testing 3.0.1-RC1-bin-20210413-f3c1468-NIGHTLY
Checking https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/
Checked https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/
Checking https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/maven-metadata.xml
Checked https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/maven-metadata.xml
19 is not a valid choice for -release
  scalac -help  gives more information
Compilation failed
Test result: 3.0.1-RC1-bin-20210413-f3c1468-NIGHTLY is a bad release

Exception in thread "main" java.lang.AssertionError: assertion failed: The evaluation script unexpectedly failed for the first checked release
        at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
        at ReleaseBisect.verifyEdgeReleases(bisect.scala:203)
        at bisect$package$.run(bisect.scala:64)
        at run.main(bisect.scala:53)

[nicolasstucki]

I assume that we have some nightly builds that are broken and cannot be used for testing. If that is the case we should try to skip them automatically. The result would be a first bad commit that is a few days after the last good commit. The commit bisection might be a bit longer in those cases but that is fine.

[WojciechMazur]

Recently I've been always limiting the scope of versions to test, eg.:

scala-cli run project/scripts/bisect.scala -- --bootstrapped --releases 3.3.1-RC1-bin-20230126-f545d10-NIGHTLY... ./bisect/script.sh

As the development of Dotty progresses there is more and more snippets that can only be compiled from more recent compiler versions.

19 is not a valid choice for -release
scalac -help gives more information

Probably Java 19 was not supported in 3.0.0. We can fix it be either using a fixed version of Java (eg. 17 which should work with all Scala 3 versions) using scala-cli directive or switching version version used locally. (I recommend sdk install/use java X.Y.Z )

I'll adapt the default script to use Java 17 by default, for any issues related with JDK18+ let's use a custom bisect with explicit java-version arguments

[WojciechMazur]

Actually I don't thing using a fixed version of JDK is a good idea, instead we can put the --jvm argument` to bisect script arguments, eg.

scala-cli project/scripts/bisect.scala -- compile --jvm=17 Test.scala

I assume that we have some nightly builds that are broken and cannot be used for testing. If that is the case we should try to skip them automatically

Can nightly builds really be broken? They're going through the CI process so we might assume they're mostly correct. However, when the do commit-based bisect we can reach invalid builds, which do not compile - I think we can try to skip these instead.

[nicolasstucki]

Can nightly builds really be broken?

Not sure. Maybe they were just not compatible with the script.