scaladoc hangs

[reid-spencer]

Compiler version

3.2.2
3.3.0-RC5
3.3.0-RC6
3.3.0
3.3.1-RC1

Minimized code

Not available yet. Extracting from 100s of classes is time-consuming.

Follow my analysis here:
https://contributors.scala-lang.org/t/scala3doc-spin-loop/6202/7

Output

None other than:

[info] Main Scala API documentation to /Users/reid/Code/reactific/riddl/language/target/scala-3.3.0/api...

Expectation

The compiler completes the documentation generation instead of hanging forever.

[reid-spencer]

To reproduce:

• Check out HEAD of main after cloning https://github.com/reactific/riddl.git
• Run sbt and issue the "doc" command

That process will never end and a java process will run at > 100% CPU utilization forever (many hours tested).

[reardonj]

I took a bit of a look at this and have a minimization. I started by getting a thread dump while generating docs with from the affected code. Scalac appears to be getting stuck in the InheritanceInformationTransformer code of scaladoc generation, performing concatenation and distinct calls on Seq. I haven't delved into the scalac code yet, but it smells like an exponential blowup when there is a large amount of inheritance. The affected code includes a deep inheritance heirarchy with a series of traits, each defining inner traits, classes, and objects. Changing the design to use self types instead allows the documentation generation to complete.

This minimization only takes several minutes instead of hanging indefinitely, but hopefully gets the idea across compactly: https://github.com/reardonj/minimize-dotty-bug-17577

"pool-77-thread-11" #610 prio=5 os_prio=31 cpu=80053.92ms elapsed=83.12s tid=0x00007fa40db38800 nid=0xaf67 runnable  [0x0000700013958000]
   java.lang.Thread.State: RUNNABLE
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.util.hashing.MurmurHash3.productHash(MurmurHash3.scala:76)
        at scala.util.hashing.MurmurHash3$.productHash(MurmurHash3.scala:343)
        at scala.runtime.ScalaRunTime$._hashCode(ScalaRunTime.scala:158)
        at dotty.tools.scaladoc.Kind$Trait.hashCode(api.scala:60)
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.util.hashing.MurmurHash3.productHash(MurmurHash3.scala:76)
        at scala.util.hashing.MurmurHash3$.productHash(MurmurHash3.scala:343)
        at scala.runtime.ScalaRunTime$._hashCode(ScalaRunTime.scala:158)
        at dotty.tools.scaladoc.LinkToType.hashCode(api.scala:141)
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.util.hashing.MurmurHash3.productHash(MurmurHash3.scala:76)
        at scala.util.hashing.MurmurHash3$.productHash(MurmurHash3.scala:343)
        at scala.Tuple2.hashCode(Tuple2.scala:24)
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.collection.mutable.HashSet.add(HashSet.scala:69)
        at scala.collection.immutable.StrictOptimizedSeqOps.distinctBy(StrictOptimizedSeqOps.scala:35)
        at scala.collection.immutable.StrictOptimizedSeqOps.distinctBy$(StrictOptimizedSeqOps.scala:26)
        at scala.collection.immutable.List.distinctBy(List.scala:79)
        at scala.collection.SeqOps.distinct(Seq.scala:206)
        at scala.collection.SeqOps.distinct$(Seq.scala:206)
        at scala.collection.AbstractSeq.distinct(Seq.scala:1188)
        at dotty.tools.scaladoc.HierarchyGraph.$plus(api.scala:146)
        at dotty.tools.scaladoc.HierarchyGraph.$plus$plus$$anonfun$1(api.scala:148)
        at dotty.tools.scaladoc.HierarchyGraph$$Lambda$6734/0x0000000801cf2040.apply(Unknown Source)
        at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
        at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
        at scala.collection.immutable.List.foldLeft(List.scala:79)
        at dotty.tools.scaladoc.HierarchyGraph.$plus$plus(api.scala:149)
        at dotty.tools.scaladoc.api$package$.withNewGraphEdges(api.scala:220)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer.apply$$anonfun$1(InheritanceInformationTransformer.scala:10)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer$$Lambda$6910/0x0000000801dca040.apply(Unknown Source)
        at dotty.tools.scaladoc.api$package$.updateRecusivly(api.scala:205)
        at dotty.tools.scaladoc.api$package$.$anonfun$1(api.scala:204)
        at dotty.tools.scaladoc.api$package$$$Lambda$6913/0x0000000801dcc840.apply(Unknown Source)
        at scala.collection.immutable.List.map(List.scala:250)
        at scala.collection.immutable.List.map(List.scala:79)
        at dotty.tools.scaladoc.api$package$.updateRecusivly(api.scala:204)
        at dotty.tools.scaladoc.api$package$.updateMembers$$anonfun$1$$anonfun$1(api.scala:240)
        at dotty.tools.scaladoc.api$package$$$Lambda$6912/0x0000000801dcb840.apply(Unknown Source)
        at scala.collection.immutable.List.map(List.scala:250)
        at scala.collection.immutable.List.map(List.scala:79)
        at dotty.tools.scaladoc.api$package$.updateMembers$$anonfun$1(api.scala:240)
        at dotty.tools.scaladoc.api$package$$$Lambda$6911/0x0000000801dcb040.apply(Unknown Source)
        at dotty.tools.scaladoc.api$package$.updatePackages(api.scala:236)
        at dotty.tools.scaladoc.api$package$.updateMembers(api.scala:240)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer.apply(InheritanceInformationTransformer.scala:10)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer.apply(InheritanceInformationTransformer.scala:5)
        at dotty.tools.scaladoc.ScalaModuleProvider$.mkModule$$anonfun$1(ScalaModuleProvider.scala:51)
        at dotty.tools.scaladoc.ScalaModuleProvider$$$Lambda$6894/0x0000000801db7840.apply(Unknown Source)
        at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
        at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
        at scala.collection.immutable.List.foldLeft(List.scala:79)
        at dotty.tools.scaladoc.ScalaModuleProvider$.mkModule(ScalaModuleProvider.scala:51)
        at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:241)
        at dotty.tools.scaladoc.Scaladoc$.run$$anonfun$1(Scaladoc.scala:73)
        at dotty.tools.scaladoc.Scaladoc$$$Lambda$6622/0x0000000801c67040.applyVoid(Unknown Source)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.Option.map(Option.scala:242)
        at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:77)
        at dotty.tools.dottydoc.Main$.process(Main.scala:25)
        at dotty.tools.dottydoc.Main.process(Main.scala)
        at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
        at xsbt.DottydocRunner.run(DottydocRunner.java:61)
        at xsbt.ScaladocInterface.run(ScaladocInterface.java:11)
        at jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
        at sbt.internal.inc.AnalyzingCompiler.invoke(AnalyzingCompiler.scala:329)
        at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:175)
        at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:133)
        at sbt.Doc$.$anonfun$scaladoc$1(Doc.scala:52)
        at sbt.Doc$.$anonfun$scaladoc$1$adapted(Doc.scala:40)
        at sbt.Doc$$$Lambda$6575/0x0000000801c35c40.apply(Unknown Source)
        at sbt.RawCompileLike$.$anonfun$prepare$1(RawCompileLike.scala:79)
        at sbt.RawCompileLike$.$anonfun$prepare$1$adapted(RawCompileLike.scala:72)
        at sbt.RawCompileLike$$$Lambda$6576/0x0000000801c36040.apply(Unknown Source)
        at sbt.RawCompileLike$.$anonfun$cached$4(RawCompileLike.scala:63)
        at sbt.RawCompileLike$.$anonfun$cached$4$adapted(RawCompileLike.scala:61)
        at sbt.RawCompileLike$$$Lambda$6584/0x0000000801c3d840.apply(Unknown Source)
        at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:219)
        at sbt.util.Tracked$$$Lambda$2940/0x0000000800db9840.apply(Unknown Source)
        at sbt.RawCompileLike$.$anonfun$cached$1(RawCompileLike.scala:68)
        at sbt.RawCompileLike$.$anonfun$cached$1$adapted(RawCompileLike.scala:52)
        at sbt.RawCompileLike$$$Lambda$6577/0x0000000801c36440.apply(Unknown Source)
        at sbt.Defaults$.$anonfun$docTaskSettings$4(Defaults.scala:2159)
        at sbt.Defaults$$$Lambda$811/0x0000000800640040.apply(Unknown Source)
        at scala.Function1.$anonfun$compose$1(Function1.scala:49)
        at scala.Function1$$Lambda$339/0x000000080040b040.apply(Unknown Source)
        at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
        at sbt.internal.util.$tilde$greater$$Lambda$2773/0x0000000800d53840.apply(Unknown Source)
        at sbt.std.Transform$$anon$4.work(Transform.scala:68)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
        at sbt.Execute$$Lambda$2799/0x0000000800d60c40.apply(Unknown Source)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
        at sbt.Execute.work(Execute.scala:291)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
        at sbt.Execute$$Lambda$2783/0x0000000800d59440.apply(Unknown Source)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
        at sbt.ConcurrentRestrictions$$anon$4$$Lambda$2796/0x0000000800d5f840.apply(Unknown Source)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
        at java.util.concurrent.FutureTask.run(java.base@11.0.11/FutureTask.java:264)
        at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.11/Executors.java:515)
        at java.util.concurrent.FutureTask.run(java.base@11.0.11/FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.11/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.11/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)