Need an option for writing scripts that are compatible across all scala3 versions

[philwalk]

This is only concerned with a way to migrate scala 3 versions to scala 3.5+ and above. An alternate title for this might be:

Provide a way to execute scripts with ".sc" extension as ".scala" files

Migrating a project with many legacy scripts requires that when bugs are encountered in production, a script can be reverted by editing the hash-bang line, and without being renamed. Also, some scripts cannot be renamed, even after the migration period, so the scala_legacy script isn't a universal option.

The need is for a way to optionally treat some .sc scripts as if they have the .scala filename extension.

Prior to scala 3.5 all scripts require a main method, so a migration format that can easily switch between old and new script semantics requires a main method.

The primary challenge is that after 3.5, the main method is not automatically called unless the file is renamed. In other words, the main method is never called if a script has the .sc extension.

Example hypothetical migration format (almost adequate, but has non-portable element on line 5 Main.main(args).

#!/usr/bin/env -S scala-cli shebang
//> using scala 3.4.3
//> using dep "com.lihaoyi::os-lib::0.11.3"
import os._
Main.main(args) // required by `3.5+`, rejected by earlier scala versions
object Main {
  def main(args: Array[String]): Unit =
    val root1 = os.root("C:\\")
    printf("cRoot: %s\n", root1)
}

Describe the solution you'd like
Possible approaches to specifying the proposed .scala mode option:

• scala-cli command line, similar to --main-class (maybe --main-method)
• system property ( "scala.dot_scala_extension")
• alternate supported extension such as .sc_ or similar, treated as if .scala
• comment //> using scala-extension-semantics

Describe alternatives you've considered

Adding support for a new filename extension .sc_ turned out to be pretty simple, and seems to work nicely, although it doesn't address the case of scripts that cannot easily be renamed. Here's the implementation:

# git diff
diff --git a/modules/build/src/main/scala/scala/build/input/Inputs.scala b/modules/build/src/main/scala/scala/build/input/Inputs.scala
index 8e93d1463..4d5b44e55 100644
--- a/modules/build/src/main/scala/scala/build/input/Inputs.scala
+++ b/modules/build/src/main/scala/scala/build/input/Inputs.scala
@@ -275,8 +275,8 @@ object Inputs {
         }
         else if path.last == Constants.projectFileName then
           Right(Seq(ProjectScalaFile(dir, subPath)))
+        else if arg.endsWith(".scala") || arg.endsWith(".sc_") then Right(Seq(SourceScalaFile(dir, subPath)))
         else if arg.endsWith(".sc") then Right(Seq(Script(dir, subPath, Some(arg))))
-        else if arg.endsWith(".scala") then Right(Seq(SourceScalaFile(dir, subPath)))
         else if arg.endsWith(".java") then Right(Seq(JavaFile(dir, subPath)))
         else if arg.endsWith(".jar") then Right(Seq(JarFile(dir, subPath)))
         else if arg.endsWith(".c") || arg.endsWith(".h") then Right(Seq(CFile(dir, subPath)))

Experiments were done based on various wrappers around scala-cli, the two most successful described here.

1. create an on-the-fly symlink in the script parent directory, but with .scala extension, which is deleted-on-exit
2. wrapper to pipe the script to scala-cli STDIN: (similar to echo 'println("Hello, Scala!")' | scala-cli -)

Approach 1 works well most of the time, but after a crash, symlinks tend to be left behind.

The pipe-to-stdin approach is equivalent to the following conversion:

jsrc/sum.sc 1 2 3 
# the equivalent command line when piped to `scala-cli` <stdin>:
tail -n +2 jsrc/sum.sc | scala-cli run -Dscript.path=jsrc/sum.sc @/opt/ue3cps _.scala -- 1 2 3

It works much of the time, but isn't viable for interactive scripts, or those that read from stdin.
The minimum startup time with this approach is about 10.0 seconds, and if procPipe.sc is compiled, about 8.0 seconds.

For comparison, the equivalent .scala version of the script starts up in less than 4 seconds, and under 1 second after it's been compiled.

Additional context
I work with projects having 1400+ legacy scripts, only about 20% have been migrated to scala 3.6.3.

[SethTisue]

I work with projects having 1400+ legacy scripts

😮 (but: cool!)

[tgodzik]

Thanks for reporting! I need to finally have a look at making classes toplevel at the generated sources. We can also add the workaround you suggested, but that would be a bandaid really.

I hope I will take a look at it this week

[philwalk]

Thanks for considering this!

We can also add the workaround you suggested, but that would be a bandaid really.

I'm not proposing the workaround, just documenting my effort to resolve this without adding a new feature.

[tgodzik]

Ok, so moving it out is more complex than I hoped, since the current IDE support relies on how the wrapper is a very simple one currently. So probably no grandiose refactor from me yet.

I have an idea how to make it work, but will be less glamorous

[tgodzik]

Got something working in #3479, though will need to wait until @Gedochao is back to see if it makes sense.

[philwalk]

Manual testing #3479 against various scripts seems to work!

[philwalk]

Most of the scripts I've tested agains #3479 work as expected. Here's the first exception.

For the following script versionSort.sc the main method doesn't get called.
It can be tested like this:

ls -l | versionSort.sc

EDIT: I just noticed that the main method parameter is _args rather than args, so perhaps this shouldn't be considered a bug. A warning might be useful, to help people that become baffled when main isn't called.

#!/usr/bin/env -S scala-cli shebang

import java.nio.file.{Path, Paths, Files}
import scala.collection.Iterator

// purpose: to sort STDIN lines numerically by subfields 
//    version_2.12/0.7.5 < version_2.12/0.7.15
object VersionSort {
  var reverse = false

  def stdin: Iterator[String] = {
    for {
      line <- Iterator.continually(scala.io.StdIn.readLine()).takeWhile( _ != null )
      if line != null
    } yield line
  }
  def main(_args: Array[String]): Unit = {
    try {
      val (ops, args) = _args.map(_.trim).partition{ (a: String) => a.startsWith("-") }
      if (ops.contains("-r")){
        reverse = true
      }
      val lines: Iterator[String] = if (args.isEmpty) {
        stdin
      } else {
        {
          for {
            line <- args.map { Paths.get(_) }.filter { Files.isRegularFile(_) }.flatMap { (p: Path) =>
              scala.io.Source.fromFile(p.toFile).getLines()
            }
          } yield line
        }.iterator
      }
      val sorted = {
        val ll = lines.map { (str: String) => PathString(str) }.toSeq.sortBy { (ps: PathString) => ps.path.toLowerCase }
        if (reverse){
          ll.reverse
        } else {
          ll
        }
      }
      for( line <- sorted ){
        printf("%s\n",line)
      }
    } catch {
    case t: Throwable =>
      t.printStackTrace
      sys.exit(1)
    }
  }
}

case class PathString(path: String) extends Ordered[PathString] {
  val fields = path.split("[\\D]+").toList.map {
    case num if num.trim.matches("[0-9]+") => "%010f".format(num.toDouble)
    case str => str
  }
  override def toString = path

  // compare 'fields'
  def compare(other: PathString): Int = {
    var i: Int = 0
    val othfld: List[String] = other.fields
    val thsfld: List[String] = this.fields
    val maxindex = othfld.size.min(thsfld.size) -1
    var num = 0
    while (i <= maxindex && num == 0) {
      num = thsfld(i) compare othfld(i)
      i += 1
    }
    num
  }
}

[tgodzik]

The name is not important though, will remove the check for it.

[tgodzik]

It should be fixed now, but let's wait for a second opinion 😅