broken support for use of wildcards in the compiler classpath on Windows

[philwalk]

Minimized code

java -Xmx20480m -Dnet.ipv6.bindv6only=0 -Dfile.encoding=UTF-8 -classpath "%CP%" -Dscala.usejavacp=true dotty.tools.dotc.Main -classpath 'c:/opt/uejlib2.13/*'  c:/opt/ue/jsrc/s3.sc

Output (click arrow to expand)

Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <*> at index 18: c:/opt/uejlib2.13/*
        at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
        at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
        at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
        at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)
        at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:229)
        at java.base/java.nio.file.Path.of(Path.java:147)
        at java.base/java.nio.file.Paths.get(Paths.java:69)
        at dotty.tools.dotc.core.MacroClassLoader$.$anonfun$1(MacroClassLoader.scala:23)
        at scala.collection.ArrayOps$.map$extension(ArrayOps.scala:924)
        at dotty.tools.dotc.core.MacroClassLoader$.makeMacroClassLoader(MacroClassLoader.scala:23)
        at dotty.tools.dotc.core.MacroClassLoader$.init(MacroClassLoader.scala:20)
        at dotty.tools.dotc.Driver.setup(Driver.scala:72)
        at dotty.tools.dotc.Driver.process(Driver.scala:192)
        at dotty.tools.dotc.Driver.process(Driver.scala:162)
        at dotty.tools.dotc.Driver.process(Driver.scala:174)
        at dotty.tools.dotc.Driver.main(Driver.scala:201)
        at dotty.tools.dotc.Main.main(Main.scala)

Use of wildcards in the classpath has been supported by java since java 6. See stack overflow: including-all-the-jars-in-a-directory-within-the-java-classpath

It has also been supported at least since scala 2.8, and is critical for working around OS command line length limitations.

The current implementation assumes that every classpath entry is a file, leading to a compiler crash when the glob entry is encountered.

An alternative way to avoid long classpath problems has been available since java 9+, and is described here (see @raman's entry):

stackoverflow: using an argument file to pass classpath entries

Briefly, it the classpath is specified in a text file (e.g., cparg), which is passed to java like so:

java @c:\path\to\cparg

A similar approach could be implemented in dotty.tools.dotc.Main.main, although ideally it would allow an os-independent format, not requiring the use of path separators (e.g., one classpath entry per line).

[smarter]

scalac @foo already exists in Scala 2 and 3 and will read any argument from the file foo

[smarter]

I've just tried it on linux and scalac -classpath 'foo/*' test.scala worked as expected, so I'm guessing this is specific to Windows? @liufengyun do you think you could investigate?

[smarter]

FWIW, the code that deals with * in classpath is https://github.com/lampepfl/dotty/blob/dbc11869127dd2e5cec205b7d64e58cf5488d2a6/compiler/src/dotty/tools/io/ClassPath.scala#L134-L142, but the crash happens in MacroClassLoader which directly passes the paths from ctx.settings.classpath to java.nio.file.Paths.get without preprocessing the *: https://github.com/lampepfl/dotty/blob/dbc11869127dd2e5cec205b7d64e58cf5488d2a6/compiler/src/dotty/tools/dotc/core/MacroClassLoader.scala#L23

[philwalk]

The problem does seem to be specific to Windows, no such problem in Linux. I will investigate further this afternoon.
By the way, in Linux I was able to do this:

scalac -classpath 'c:/opt/uejlib2.13/*' /opt/ue/jsrc/s3.sc

The invocation in Windows that I provided above is basically the java command that resulted when I issued the same command from a cygwin session, so it's possible that the "scalac" bash script behaves incorrectly in Windows. Is there a plan to provide Windows batch files? To be clear, I'm not advocating the need for scalac.bat, etc., but am curious.

[smarter]

I don't think the script is the source of the problem, the problem is we pass wildcards to java.nio.file.Paths.get which happens to work on Linux but not Windows.

[philwalk]

When I reduce the code in dotty/compiler/src/dotty/tools/dotc/core/MacroClassLoader.scala line 23, to the following script:

val cp = "/opt/uejlib2.13/*"
val url = java.nio.file.Paths.get(cp).toUri.toURL
println(url)

I get different behavior in Windows versus Linux:

Windows:

java.nio.file.InvalidPathException: Illegal char <*> at index 16: /opt/uejlib2.13/*
        at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
        at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
        at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
        at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)
        at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:229)
        at java.base/java.nio.file.Path.of(Path.java:147)
        at java.base/java.nio.file.Paths.get(Paths.java:69)
        at Main$$anon$1.<init>(starTest.sc:4)
        at Main$.main(starTest.sc:2)
        at Main.main(starTest.sc)
        [snip remaining stack dump]

Linux:

file:/opt/uejlib2.13/*

So java.nio.file.Path treats a wildcard entry differently in Windows.

This seems to explain why:

https://stackoverflow.com/questions/27522581/asterisks-in-java-path

The short version: asterisk is a legal filename character in Linux, but not in Windows.

I will investigate using the scalac @foo approach.

[philwalk]

FYI, the compiler doesn't currenty permit source files to be on a drive other than the current session drive.

My /tmp directory is mounted to G: (a disk drive) to avoid wearing out my SSD drive.
When I try to compile a temporary copy of my scala script (with the hash-bang line removed), I get this:

/opt/scala3/bin/scalac @.uejlib213Classpath "G:/tmp/s3.sc"

Output (click arrow to expand)

exception occurred while compiling G:\tmp\s3.sc

java.lang.IllegalArgumentException: 'other' has different root while compiling G:/tmp/s3.sc
Exception in thread "main" java.lang.IllegalArgumentException: 'other' has different root
        at java.base/sun.nio.fs.WindowsPath.relativize(WindowsPath.java:404)
        at java.base/sun.nio.fs.WindowsPath.relativize(WindowsPath.java:42)
        at dotty.tools.dotc.transform.PostTyper$PostTyperTransformer.transform(PostTyper.scala:346)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform$$anonfun$2(Trees.scala:1369)
        at scala.collection.immutable.List.mapConserve(List.scala:472)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1369)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transformStats(Trees.scala:1367)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1354)
        at dotty.tools.dotc.transform.MacroTransform$Transformer.transform(MacroTransform.scala:54)
        at dotty.tools.dotc.transform.PostTyper$PostTyperTransformer.transform(PostTyper.scala:417)
        at dotty.tools.dotc.transform.MacroTransform.run(MacroTransform.scala:21)
        at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:296)
        at scala.collection.immutable.List.map(List.scala:246)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:297)
        at dotty.tools.dotc.Run.runPhases$4$$anonfun$4(Run.scala:185)
        at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
        at dotty.tools.dotc.Run.runPhases$5(Run.scala:195)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:203)
        at dotty.runtime.function.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:67)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:210)
        at dotty.tools.dotc.Run.compileSources(Run.scala:147)
        at dotty.tools.dotc.Run.compile(Run.scala:129)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:38)
        at dotty.tools.dotc.Driver.process(Driver.scala:193)
        at dotty.tools.dotc.Driver.process(Driver.scala:162)
        at dotty.tools.dotc.Driver.process(Driver.scala:174)
        at dotty.tools.dotc.Driver.main(Driver.scala:201)
        at dotty.tools.dotc.Main.main(Main.scala)
compile error

A workaround is to pushd g:/tmp, compile, and then popd.

[liufengyun]

@philwalk This is a bug (#10763). Before we fix that, a workaround is to pass the option -sourceroot G:/tmp.