script with java.sql.Date reference hits a NoClassDefFoundError when running from compiled jar

[philwalk]

Compiler version

Scala code runner version 3.1.1-RC2 -- Copyright 2002-2021, LAMP/EPFL

The problem occurs when running a script with these runtime java versions:

• openjdk version "16.0.1" 2021-04-20
• java version "11.0.12" 2021-07-20 LTS

The problem does NOT occur with java 8

• openjdk version "1.8.0_312"

The symptoms are similar to those reported in #11646, although the problem here is restricted to running the compiled jar.

Minimized code

If a script references java.sql.Date, a NoClassDefFoundError occurs, but only when running from a compiled script jar.

Here's the test script (notice the -save option in the hash bang line):

#!/opt/scala3/bin/scala -save
def main(args: Array[String]): Unit = {
    val d: java.sql.Date = new java.sql.Date(100L) // Compile and execution is successfully.
    println(d)
}

Output

The compile-and-run pass is successful:

/opt/scala3/bin/scala -save sqlDateError.sc
C:/opt/scala3

But because -save is specified, subsequent script runs execute from the compiled jar file, producing the error java.lang.NoClassDefFoundError: java/sql/Date

$ /opt/scala3/bin/scala -save ./sqlDateBug.sc

Exception in thread "main" java.lang.NoClassDefFoundError: java/sql/Date
        at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
        at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
        at java.base/java.lang.Class.getMethodsRecursive(Class.java:3307)
        at java.base/java.lang.Class.getMethod0(Class.java:3293)
        at java.base/java.lang.Class.getMethod(Class.java:2106)
        at dotty.tools.runner.RichClassLoader$.run$extension(ScalaClassLoader.scala:91)
        at dotty.tools.runner.CommonRunner.run(ObjectRunner.scala:23)
        at dotty.tools.runner.CommonRunner.run$(ObjectRunner.scala:13)
        at dotty.tools.runner.ObjectRunner$.run(ObjectRunner.scala:48)
        at dotty.tools.runner.CommonRunner.runAndCatch(ObjectRunner.scala:30)
        at dotty.tools.runner.CommonRunner.runAndCatch$(ObjectRunner.scala:13)
        at dotty.tools.runner.ObjectRunner$.runAndCatch(ObjectRunner.scala:48)
        at dotty.tools.MainGenericRunner$.run$1(MainGenericRunner.scala:218)
        at dotty.tools.MainGenericRunner$.main(MainGenericRunner.scala:238)
        at dotty.tools.MainGenericRunner.main(MainGenericRunner.scala)
Caused by: java.lang.ClassNotFoundException: java.sql.Date
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 15 more
``` 

However, there seems to be nothing wrong with the compiled jar itself, which can be run directly by java:

java -jar sqlDateBug.jar

Expectation

A script running from the compiled jar file should behave the same as when doing a compile-and-run.

[philwalk]

A partial solution to this problem is mentioned here: jdk9 removal of common profiles

Manually adding the following attribute to MANIFEST.MF of the compiled jar seems to be part of the solution:
Add-Exports: java.sql

It turns out that the compiled jar is executable without the Add-Exports, or Add-Modules, or other attributes if called via java -jar.

java -jar sqlDateError.jar

This also requires changes to dotty.tools.MainGenericRunner and dotty.tools.scripting.Main.

We could, by default, add the modules list in the compact3 profile of jdk9, which was removed in later jdk versions. As described in the link above, that would include the following modules:

• java.base, java.logging, java.scripting, java.rmi, java.sql, java.xml, java.compiler, java.instrument, java.management, java.naming, java.prefs, java.security.jgss, java.security.sasl, java.sql.rowset, java.xml.crypto

[michelou]

@philwalk The section Removed Tools Support for Compact Profiles you mention in your issue description also states the following :

"[...] You can use the jdeps tool to do a static analysis of the Java packages that are being used in your source code."

When targetting JDK 9+ a library developer (e.g. authoring sqlDateError.jar) should be aware of the usage of compact profiles and should take care appropriately.

As a first reaction I would say it's not a good idea to modify dotty.tools.MainGenericRunner and dotty.tools.scripting.Main.

PS. Based on my experience (I'm coding in Java since JDK 1.O) people must accept that developping under JDK 9+ is a more demanding task.

[philwalk]

As a first reaction I would say it's not a good idea to modify dotty.tools.MainGenericRunner and dotty.tools.scripting.Main

Changes that are necessary haven't yet been registered in a bug report, but there is an issue that needs to be addressed:

• the wrong classpath is used when executing from a jar file

When testing the minimized code above (with scala3-3.1.1-RC2):

• the compile and run phase is successful,
• the run from the compiled jar throws java.lang.NoClassDefFoundError: java/sql/Date

However, when running the compiled jar directly via java -jar sqlDateError.jar, it quietly catches and discards the exception. it runs successfully, although the original version of the script would not print anything, since sys.props("scala.home") would not be initialized in that case.

[michelou]

@philwalk Thanks for the clarification. I can't wait to see the proposed changes and their effect on runtime behavior 😉

[philwalk]

Updating the classpath when dotty.tools.MainGenericRunner runs from the Jar file does not resolve this issue, so I will continue to investigate.

Also note the simplified test script above. It can be expected to behave the same way whether the jar is executed internally by dotty.tools.MainGenericRunner or directly via java -jar

The original script would need -Dscala.home=/opt/scala3 or similar on the java -jar command line in order to print anything.

To my surprise, the problem also occurs with java 8, so it seems to have nothing to do with the java module system of jdk9+.
I wasn't able to duplicate the error with java 8, so my best guess is that this does seem to be related to jdk9+ modules.

[philwalk]

#14241 provides 2 fixes plus a workaround for this bug.

• exceptions are always reported when executing from a compiled jar file
• will not execute from a previously compiled jar file unless the -save option is specified
• a -nosave option provides a way to selectively prevent scripts affected by this bug from executing via the jar

[BarkingBad]

@philwalk should we close it or is you want to continue work on this issue?

[philwalk]

should we close it or is you want to continue work on this issue?

I don't know if we should close it, the underlying problem is still there, and can possibly be fixed the same way #11658 was fixed, by using a different classloader. However, I won't be able to work on it soon.