StackOverflowError when using Java code that creates types with LazyRef

[wjoel]

Compiler version

3.5.0, works on 3.4.2, 3.4.3.

Minimized code

I have unfortunately not been able to minimize this to a self contained example.

//> using scala 3.5.0
//> using dep io.github.jwharm.javagi:gtk:0.10.2

import org.gnome.gtk.{Unit as _, *}

object Main {
  Box.builder()
    .setOrientation(Orientation.VERTICAL)
    .setHalign(Align.START)
    .setValign(Align.BASELINE)
    .se
    .build()
    .asInstanceOf[Box]
}

Note that .se (which does not exist, and should be a compiler error) is intentional, and triggers the stack overflow.
Scala 3.4.2 fails with a reasonable error message:

 7 |  Box.builder()
 8 |    .setOrientation(Orientation.VERTICAL)
 9 |    .setHalign(Align.START)
10 |    .setValign(Align.BASELINE)
11 |    .se
   |  ^
   |  value se is not a member of org.gnome.gtk.Box#Builder[? <: org.gnome.gtk.Box#Builder[?]]#B#B² - did you mean B.ne?
   |
   |  where:    B  is a type in trait Builder² with bounds <: io.github.jwharm.javagi.gobject.Builder³[B]
   |            B² is a type in class Builder⁴ with bounds <: org.gnome.gtk.Widget#Builder⁴[B²]

Output (click arrow to expand)

  Exception while compiling /home/wjoel/dev/scala3-playground/Box.java, /home/wjoel/dev/scala3-21:47:04 [1004/45939]
, /home/wjoel/dev/scala3-playground/Orientable.java, /home/wjoel/dev/scala3-playground/Widget.java

  An unhandled exception was thrown in the compiler.
  Please file a crash report here:
  https://github.com/scala/scala3/issues/new/choose
  For non-enriched exceptions, compile with -Xno-enrich-error-messages.

     while compiling: <no file>
        during phase: parser
                mode: Mode()
     library version: version 2.13.14
    compiler version: version 3.5.0
            settings: -classpath /home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-l
ibrary_3/3.5.0/scala3-library_3-3.5.0.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwha
rm/javagi/gtk/0.10.2/gtk-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala
-library/2.13.14/scala-library-2.13.14.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwh
arm/javagi/gdk/0.10.2/gdk-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/ja
vagi/gsk/0.10.2/gsk-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/g
dkpixbuf/0.10.2/gdkpixbuf-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/ja
vagi/gio/0.10.2/gio-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/p
ango/0.10.2/pango-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/pan
gocairo/0.10.2/pangocairo-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/ca
irobindings/cairo/1.18.3/cairo-1.18.3.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwha
rm/javagi/graphene/0.10.2/graphene-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/
jwharm/javagi/gmodule/0.10.2/gmodule-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/githu
b/jwharm/javagi/gobject/0.10.2/gobject-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/git
hub/jwharm/javagi/harfbuzz/0.10.2/harfbuzz-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io
/github/jwharm/javagi/glib/0.10.2/glib-0.10.2.jar -d /home/wjoel/dev/scala3-playground/.scala-build/scala3-playgroun
d_1a8a7b68eb/classes/main -java-output-version 22 -sourceroot /home/wjoel/dev/scala3-playground

Exception in thread "main" java.lang.StackOverflowError
        at dotty.tools.dotc.typer.ImplicitRunInfo$collectParts$2$.traverse(Implicits.scala:681)
        at dotty.tools.dotc.typer.ImplicitRunInfo$collectParts$2$.apply(Implicits.scala:686)
        at dotty.tools.dotc.typer.ImplicitRunInfo.recur$1(Implicits.scala:762)
        at dotty.tools.dotc.typer.ImplicitRunInfo.iscopeRefs$1(Implicits.scala:706)
        at dotty.tools.dotc.typer.ImplicitRunInfo.addCompanions$1(Implicits.scala:739)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1$$anonfun$1(Implicits.scala:757)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.mutable.LinkedHashSet.foreach(LinkedHashSet.scala:128)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1(Implicits.scala:757)
        at dotty.tools.dotc.typer.ImplicitRunInfo.recur$1(Implicits.scala:763)
        at dotty.tools.dotc.typer.ImplicitRunInfo.iscopeRefs$1(Implicits.scala:706)
        at dotty.tools.dotc.typer.ImplicitRunInfo.addCompanions$1(Implicits.scala:739)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1$$anonfun$1(Implicits.scala:757)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.mutable.LinkedHashSet.foreach(LinkedHashSet.scala:128)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1(Implicits.scala:757)
        at dotty.tools.dotc.typer.ImplicitRunInfo.recur$1(Implicits.scala:763)
        at dotty.tools.dotc.typer.ImplicitRunInfo.iscopeRefs$1(Implicits.scala:706)
        at dotty.tools.dotc.typer.ImplicitRunInfo.addCompanions$1(Implicits.scala:739)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1$$anonfun$1(Implicits.scala:757)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.mutable.LinkedHashSet.foreach(LinkedHashSet.scala:128)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1(Implicits.scala:757)
        at dotty.tools.dotc.typer.ImplicitRunInfo.recur$1(Implicits.scala:763)
        at dotty.tools.dotc.typer.ImplicitRunInfo.iscopeRefs$1(Implicits.scala:706)
        at dotty.tools.dotc.typer.ImplicitRunInfo.addCompanions$1(Implicits.scala:739)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1$$anonfun$1(Implicits.scala:757)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.mutable.LinkedHashSet.foreach(LinkedHashSet.scala:128)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1(Implicits.scala:757)
        at dotty.tools.dotc.typer.ImplicitRunInfo.recur$1(Implicits.scala:763)
        at dotty.tools.dotc.typer.ImplicitRunInfo.iscopeRefs$1(Implicits.scala:706)
        at dotty.tools.dotc.typer.ImplicitRunInfo.addCompanions$1(Implicits.scala:739)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1$$anonfun$1(Implicits.scala:757)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.mutable.LinkedHashSet.foreach(LinkedHashSet.scala:128)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1(Implicits.scala:757)
        at dotty.tools.dotc.typer.ImplicitRunInfo.recur$1(Implicits.scala:763)
        at dotty.tools.dotc.typer.ImplicitRunInfo.iscopeRefs$1(Implicits.scala:706)
        at dotty.tools.dotc.typer.ImplicitRunInfo.addCompanions$1(Implicits.scala:739)
        at dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1$$anonfun$1(Implicits.scala:757)
... and so on

[EugeneFlesselle]

Myself and @hamzaremmal were not able to reproduce the issue. In any of the 4 configurations with either Scala 3.5.0 or the latest nightly, and with either Java 17 or Java 22; we get the following error:

-- [E008] Not Found Error: /Users/eugeneflesselle/Documents/dotty/tests/playground/example.scala:11:5 
 7 |  Box.builder()
 8 |    .setOrientation(Orientation.VERTICAL)
 9 |    .setHalign(Align.START)
10 |    .setValign(Align.BASELINE)
11 |    .se
   |  ^
   |value se is not a member of org.gnome.gtk.Box#Builder[? <: org.gnome.gtk.Box#Builder[?]]#B#B².
   |Extension methods were tried, but the search failed with:
   |
   |    Recursion limit exceeded.
   |    Maybe there is an illegal cyclic reference?
   |    If that's not the case, you could also try to increase the stacksize using the -Xss JVM option.
   |    For the unprocessed stack trace, compile with -Xno-decode-stacktraces.
   |    A recurring operation is (inner to outer):
   |    
   |      collectParts of org.gnome.gtk.Widget#Builder[
   |      org.gnome.gtk.Box#Builder[? <: org.gnome.gtk.Box#Builder[?]]#B#B]
   |      collectParts of  <:
   |      org.gnome.gtk.Widget#Builder[
   |        org.gnome.gtk.Box#Builder[? <: org.gnome.gtk.Box#Builder[?]]#B#B]
   |
   |where:    B  is a type in trait Builder² with bounds <: io.github.jwharm.javagi.gobject.Builder³[B]
   |          B² is a type in class Builder⁴ with bounds <: org.gnome.gtk.Widget#Builder⁴[B²]

The stack trace in the original issue points to code using handleRecursive whose goal is precisely to catch stack overflows caused by theses cyclic references. I'm not sure how the unprocessed stacks trace got reported without the -Xno-decode-stacktraces option.

[wjoel]

Thanks for looking into this, @EugeneFlesselle! That is odd. It shouldn't matter, but I'm on Linux, Fedora 40.

$ scala-cli -version
Scala CLI version: 1.5.0
Scala version (default): 3.5.0
$ java -version
openjdk version "22.0.2" 2024-07-16
OpenJDK Runtime Environment Zulu22.32+15-CA (build 22.0.2+9)
OpenJDK 64-Bit Server VM Zulu22.32+15-CA (build 22.0.2+9, mixed mode, sharing)

And with this in reprod-i21521.sh

#!/bin/bash

mkdir i21521
cd i21521
echo '//> using scala 3.5.0
//> using dep io.github.jwharm.javagi:gtk:0.10.2

import org.gnome.gtk.{Unit as _, *}

object Main {
  Box.builder()
    .setOrientation(Orientation.VERTICAL)
    .setHalign(Align.START)
    .setValign(Align.BASELINE)
    .se
    .build()
    .asInstanceOf[Box]
}' > main.scala && scala-cli compile .

I get:

wjoel@apollo:~/dev$ ./reprod-i21521.sh 
Compiling project (Scala 3.5.0, JVM (22))
Error: Encountered a StackOverflowError coming from the compiler. You might need to restart your Bloop build server:
dotty.tools.dotc.typer.ImplicitRunInfo$collectParts$2$.traverse(Implicits.scala:681)
dotty.tools.dotc.typer.ImplicitRunInfo$collectParts$2$.apply(Implicits.scala:686)
dotty.tools.dotc.typer.ImplicitRunInfo.recur$1(Implicits.scala:762)
dotty.tools.dotc.typer.ImplicitRunInfo.iscopeRefs$1(Implicits.scala:706)
dotty.tools.dotc.typer.ImplicitRunInfo.addCompanions$1(Implicits.scala:739)
dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1$$anonfun$1(Implicits.scala:757)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
scala.collection.mutable.LinkedHashSet.foreach(LinkedHashSet.scala:128)
dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1(Implicits.scala:757)
dotty.tools.dotc.typer.ImplicitRunInfo.recur$1(Implicits.scala:763)
dotty.tools.dotc.typer.ImplicitRunInfo.iscopeRefs$1(Implicits.scala:706)
dotty.tools.dotc.typer.ImplicitRunInfo.addCompanions$1(Implicits.scala:739)
dotty.tools.dotc.typer.ImplicitRunInfo.collectCompanions$1$$anonfun$1(Implicits.scala:757)
...

Restarting bloop (it wasn't actually started when I ran this the first time, but just in case, I tried again after scala --power bloop exit) does not make any difference.

That being said, there might be a (false) cyclic reference hidden in the LazyRefs. I say false, because it works with other versions of Scala 3, and it works in Java. Does my PR take care of your supposedly cyclic reference and stack overflow being reported?

[EugeneFlesselle]

@wjoel Ah! I was able to reproduce it, thank you for the additional details. Could you try it with bloop disabled completely --server=false ? It prevents the overflow from occurring on my end.

Does my PR take care of your supposedly cyclic reference and stack overflow being reported?

It does, I'll add a review over on the PR directly.

[wjoel]

I still get an overflow, it's just printed differently (no suggestion to restart the bloop server, for example, and with more details):

  Exception while compiling /home/wjoel/dev/i21521/main.scala

  An unhandled exception was thrown in the compiler.
  Please file a crash report here:
  https://github.com/scala/scala3/issues/new/choose
  For non-enriched exceptions, compile with -Xno-enrich-error-messages.

     while compiling: <no file>
        during phase: parser
                mode: Mode()
     library version: version 2.13.14
    compiler version: version 3.5.0
            settings: -classpath /home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.5.0/scala3-library_3-3.5.0.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/gtk/0.10.2/gtk-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.14/scala-library-2.13.14.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/gdk/0.10.2/gdk-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/gsk/0.10.2/gsk-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/gdkpixbuf/0.10.2/gdkpixbuf-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/gio/0.10.2/gio-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/pango/0.10.2/pango-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/pangocairo/0.10.2/pangocairo-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/cairobindings/cairo/1.18.3/cairo-1.18.3.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/graphene/0.10.2/graphene-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/gmodule/0.10.2/gmodule-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/gobject/0.10.2/gobject-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/harfbuzz/0.10.2/harfbuzz-0.10.2.jar:/home/wjoel/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/jwharm/javagi/glib/0.10.2/glib-0.10.2.jar -d /home/wjoel/dev/i21521/.scala-build/i21521_cef53d4a2d/classes/main -sourceroot /home/wjoel/dev/i21521

Exception in thread "main" java.lang.StackOverflowError
	at dotty.tools.dotc.typer.ImplicitRunInfo$collectParts$2$.traverse(Implicits.scala:681)
	at dotty.tools.dotc.typer.ImplicitRunInfo$collectParts$2$.apply(Implicits.scala:686)
	at dotty.tools.dotc.typer.ImplicitRunInfo.recur$1(Implicits.scala:762)
	at dotty.tools.dotc.typer.ImplicitRunInfo.iscopeRefs$1(Implicits.scala:706)
...

[EugeneFlesselle]

With --server=false appended to reprod-i21521.sh, I consistently get the cyclic reference error message, on MacOS with Scala 3.5.0, and Java 17. @Gedochao Is anyone on another system (still with bloop disabled) able to reproduce the issue ?

[wjoel]

I've bisected this, and the bad commit is 9d88c80, "Drop redundant butNot = Param clause in isAnchor". Undoing that,

modified   compiler/src/dotty/tools/dotc/typer/Implicits.scala
@@ -636,7 +636,7 @@ trait ImplicitRunInfo:
   private def isAnchor(sym: Symbol) =
     sym.isClass && !isExcluded(sym)
     || sym.isOpaqueAlias
-    || sym.is(Deferred)
+    || sym.is(Deferred, butNot = Param)
     || sym.info.isMatchAlias
 
   private def computeIScope(rootTp: Type): OfTypeImplicits =

... I get the same error message as on 3.4.2. So I'm guessing that butNot = Param is in fact not redundant, and the removal causes some LazyRefs to be created where they shouldn't?