Scala distribution replacement proposal

[adriaanm]

By @dwijnand

This proposal seeks to address the problem of new users installing Scala, either with their package manager or with the distribution archives, and getting a setup that doesn't match how the majority of the Scala community consumes and develops in Scala.

The proposed way to address this is by:

1. Removing the Scala distribution archives from the Scala website (and, therefore, from the Scala project's build and release setup and procedure docs.)
2. Ensure sbt is appropriately highlighted and documented as the primary way the Scala ecosystem consumes Scala and builds their Scala libraries and apps, with sufficient detail on how to install and get started with sbt for it to replace the Scala distribution for new users learning and starting to develop in Scala.
3. Make strides to ensure that there is a continuation path for the operating system-specific archives present in the Scala distribution, such as the deb, rpm and msi files generated.
4. Provide a replacement solution for the legitimate users of the command-line runners in the Scala distribution (scala, scalac, scaladoc, and scalap).

Note that Scala will continue to be distributed as jars to Maven Central, which is how the majority of the ecosystem consumes Scala. A side-benefit of this change would, therefore, be that it would simplify, and thus reduce the maintainance cost of, the Scala project's build and release procedure.

A secondary action that is also proposed is to:

• Somewhat promote and bring attention to Coursier and its app installing and launching capabilities, as a way to gain access to other tools in the Scala ecosystem, outside of what might already be available with their package manager, in order to promote (a) the command-line runners replacement, (b) other installable apps of primary interest for new users, such as for Creative Scala or Scala 3, perhaps.

Background

Historically the Scala distribution has been available as: (in addition to api docs and sources archives)

• a tarball (.tgz)
• a zip file
• a msi installer file
• a deb package file
• a rpm package file

Within the distribution are the Scala projects jars, and any dependencies:

• scala-library
• scala-reflect
• scala-compiler
• scalap
• jline

as well as the following runner scripts, as both bash and BAT scripts, presented here with added description of what they allow users to do from the command-line):

• scala: a multifaceted Scala runner, that allows:
  • scala Main.scala to run a Scala "script", that is compile and run a Scala source file, possibly in a synthetic main method with a synthetic args,
  • scala Main to run a Scala class with a main method,
  • scala main.jar to run a Scala jar with a Main-Class,
  • scala to run the Scala REPL,
• scalac: the (batch) Scala compiler,
• scaladoc: the Scaladoc tool,
• scalap: the Scala class file disassembler, and lastly
• fsc: the "fast" Scala compiler, which is disrecommended

with optionally the ability to define any:

• Scala compiler options,
• Java system properties,
• Java options, and
• Java classpath (hand-constructed)

Many years ago, when bootstrapping the Scala language and its subsequent ecosystem, that worked well. However, nowadays the majority of Scala users working on Scala projects will mostly never use the command-line runners or the bundled jars in the Scala distribution. A large number of Scala users will instead mostly interact with sbt, the most commonly used build tool in the Scala community, to:

• run their Scala project,
• run the Scala REPL within their project,
• resolve their project's dependencies, defining their project's classpath, and
• compile, test, generate scaladoc, package, and publish or distributed their Scala projects.

A lot of tools in the Scala ecosystem provide an integration with sbt in the form of an sbt plugin which users consume by modifying either their build definition or their user-wide sbt settings. However, not every tool in a Scala user's toolkit is strictly part their project's build. So having every Scala tool be accessed through sbt is an avoidable imposition.

Unfortunately, within the larger JVM community there hasn't been any popular, general-purpose tool for downloading, launching, or bootstrapping tools, which has lead the ecosystem to repeatedly creating and maintaining a tangle of bash, BAT scripts and installation docs, with their related system properties, environment variables, and configuration files.

Within the Scala ecosystem, over time, there have been several, disconnected, projects that seeked to address how to launch, and possibly boostrap and/or install, command-line tools in the Scala ecosystems:

• sbaz, the "Scala Bazaar System" project, a package manager for Scala installations, now dead,
• sbt/launcher, a module of sbt which, while initially built for sbt, was made to be a general-purpose Scala application jar and dependency jar downloader and launcher, but really only used for the sbt project to dynamically download and launch the version of sbt defined by the sbt.version key in project/build.properties
• Conscript, a distribution mechanism for Scala apps, built using sbt/launcher, now dead; and lastly
• Coursier, initially just a pure-Scala replacement dependency manager to sbt's use of Ivy, but which later also developed general-purpose Scala application launching, bootstrapping, and installing capabilities

Some of the most popular tools built in Scala and used by the Scala ecosystem, and how they run, are:

• the Scala runner scripts, as mentioned above, that rely on the existence of the library jars and dependencies on the filesystem,
• sbt, Scala's build tool, which actually has two runners:
  • the official sbt runner scripts in sbt-launcher-package, as both bash and BAT scripts, which noteably relies on the presence of sbt's sbt-launch.jar (which is based on sbt/launcher) on the filesystem,
  • the unofficial, but fairly used, sbt-extras runner script, as a UNIX and Cgywin bash script only, which noteably is used on Travis CI and by some Scala compiler engineers, and also noteably dynamically downloads the sbt's launch jar meaning all it needs is the runner script (and Java),
• The old "Typesafe Zinc", which provided sbt's incremental compiler module for consumption by other build systems (nowadays Zinc refers to the module itself, as an independent project),
• Ammonite, an alternative Scala REPL, which isn't based on any general purpose installers or launcher,
• Almond, which brings Scala support to Jupyter's notebooks and uses Coursier (and Ammonite),
• Scalafmt, a popular code formatter for Scala, that uses Coursier,
• Scalafix, a refactoring and linting tool for Scala, that also uses Coursier, and finally
• Bloop, a Scala build server, that also uses Coursier behind a Python script veneer
• Mill, an alternative build tool for Scala, like Ammonite doesn't use anything general purpose
• Fury, another alternative build tool for Scala, which uses a bash script veneer over Coursier,

Of additional importance sbt's launcher and sbt use a custom, non-standard classloader setup in order, for example, for the sbt launcher to be written in Scala 2.10 but run sbt 1 which is written in scala 2.12. See Jason Zaugg's notes for more details.

Notes and Design Decisions

• scala-runners is a proposed replacement for the runner scripts, based on coursier launch
• what if package maintainers just bundle scala-runners? That would undermine everything.
• ok to drop fsc?
• additional coursier app definitions?
  • scala-repl, for just the official Scala REPL? (or called scalar?)
  • creative-scala, for non-programmer beginners,
  • dotty
  • dotty-repl/dotr
  • ammonite
• the sbt runner will use the latest stable version of Coursier and sbt, at the time of the Scala distribution
• see Rustup for comparisions and inspiration on a language installer, manager and distribution
• no download, usage, "phone home" statistics gathering (see Ammonite's controversy)
• dropping debs and rpms: everything OK to let the Debian/Ubuntu and RedHat communities maintain them externally?
• drop MSI: everything OK to let the Chocolately and/or Scoop communities maintain it externally?
• drop sbt's distribution archives? what about preloaded dependencies?
• coursier-based sbt launching
• there will be overlap with OS package manager packages/formula, e.g. a separate coursier formula
• legal concerns with Lightbend or EPFL or the Apache 2 license with dropping the distribution?

Participants

Lead: Dale Wijnand (Scala contributor, sbt ex-maintainer)

I propose we invite the people we know in the community have exhibited an interest in this space, and might be interested in being involved:

• Alexandre Archambault - Coursier
• Eugene Yokota - sbt
• Jason Zaugg - sbt launcher classloader notes and ideas
• Jorge Vicente Cantero - sbt and Bloop
• Jon Pretty - Fury
• Kenji Yoshida - Conscript
• Li Haoyi - Ammonite and Mill
• Ólafur Páll Geirsson - Scalafmt and Scalafix

Let me know if you have any other suggestions.

[jvican]

Interesting! /CC @heathermiller

[antonkulaga]

I personally even do not bother installing Scala, as sbt is enough to make Scala projects work and for REPL there is ammonite-shell

[retronym]

I've cataloged a few ideas for simplifying the SBT launcher, in particular around its classloader hierarchy. I'll work through these with @eed3si9n and @dwijnand, keeping the context of consolidation in mind.

[jvican]

@retronym I am actually working on a smarter mechanism to cache classloaders for sbt plugins, it builds upon the same concepts you describe in the catalogue. Will submit PR soon.

[curoli]

I routinely download the DEB or RPM files from the website to install Scala on all of my Linux machines. Is it really much work to provide those? I created DEB files of my own apps by SBT Native Packager and it has been quick and easy.