Compilation order dependencies in compileStdLib

[milessabin]

Running testCompilation locally fails for me (Linux, Oracle JDK 1.8.0_192) due to what appears to be a compilation order dependency in compileStdLib. This isn't normally visible in CI runs, presumably due to platform dependency of the compilation order. Sorting the sources, as done in #5432 causes local and CI builds to fail in the same way.

[milessabin]

After updating I no longer see a local failure unless I explicitly sort the sources, so local and CI appear to be on a par now.

I've attempted to minimize the problem and come up with the following,

dotc -language:Scala2 -d local/classes \
  scala2-library/src/library/scala/collection/IterableViewLike.scala \
  scala2-library/src/library/scala/collection/SeqViewLike.scala \
  scala2-library/src/library/scala/collection/TraversableViewLike.scala \
  scala2-library/src/library/scala/collection/immutable/Stream.scala

Compiling the three ViewLikes on their own, or the Stream on it's own completes successfully.

However, with all four the compiler crashes with,

***** missing reference, looking for Transformed/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Transformed/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for AbstractTransformed/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to AbstractTransformed/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for EmptyView/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to EmptyView/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for Forced/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Forced/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for Sliced/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Sliced/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for Mapped/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Mapped/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for FlatMapped/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to FlatMapped/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for Appended/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Appended/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for Filtered/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Filtered/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for TakenWhile/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to TakenWhile/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for DroppedWhile/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to DroppedWhile/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for Zipped/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Zipped/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for ZippedAll/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to ZippedAll/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for Reversed/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Reversed/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for Patched/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Patched/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).
***** missing reference, looking for Prepended/T in trait SeqViewLike
decls = Scope{ ... }
Bad symbolic reference. A signature in /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class)
refers to Prepended/T in trait scala.collection.SeqViewLike which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling /home/miles/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.7.jar(scala/collection/immutable/StreamViewLike.class).

[smarter]

This seems like a different issue: to compile Stream we unpickle StreamViewLike from a Scala 2 classfile, which then tries to find some symbol in SeqViewLike that we're also trying to compile. This sort of Dotty->Scala2->Dotty dependency chain is not something that should ever happen in a real project (that is, until we make Scala 2 use Dotty as a backend) so we shouldn't try to make it work.

If you add StreamViewLike to the list of files you're compiling, do you still get a crash ?

[milessabin]

This sort of Dotty->Scala2->Dotty dependency chain is not something that should ever happen in a real project (that is, until we make Scala 2 use Dotty as a backend) so we shouldn't try to make it work.

Perhaps. But then removing any source file from the full compileStdLib set is going to involve unpickling the missing items from the Scala 2 classpath. If that's the case then it looks like any attempt to minimize the original problem will change the nature of the issue.

[milessabin]

If you add StreamViewLike to the list of files you're compiling, do you still get a crash ?

No it doesn't.

[smarter]

If that's the case then it looks like any attempt to minimize the original problem will change the nature of the issue.

One way around that would be to first change the top level package (easiest way to do it is to add "package foo" at the top of every file) then do the minimization.

[milessabin]

One way around that would be to first change the top level package (easiest way to do it is to add "package foo" at the top of every file) then do the minimization.

TBQH, that's what I think the full test really ought to be.

[smarter]

Well we do need to test the actual standard library as we would compile it as part of the bootstrapping process, but since we're moving to the 2.13 library anyway, it'd make sense to import the sources of the 2.12 library in the dotty repo, add an extra top package, and make it a regular test.

[allanrenucci]

Reduced to 4 files:

dotc -Ycheck:all -language:Scala2 scala2-library/src/library/scala/collection/IterableViewLike.scala scala2-library/src/library/scala/collection/SeqViewLike.scala scala2-library/src/library/scala/collection/TraversableViewLike.scala scala2-library/src/library/scala/collection/mutable/IndexedSeqView.scala

[allanrenucci]

And here is a single file reproduction:

package scala.collection

trait IterableViewLike[+A, +Coll, +This <: IterableView[A, Coll] with IterableViewLike[A, Coll, This]]
     extends TraversableView[A, Coll] {

  trait TransformedI[+B] extends IterableView[B, Coll] with super.TransformedT[B]
}

trait SeqViewLike[+A, +Coll, +This <: SeqView[A, Coll] with SeqViewLike[A, Coll, This]]
  extends IterableView[A, Coll] with IterableViewLike[A, Coll, This] {

  trait TransformedS[+B] extends SeqView[B, Coll] with super.TransformedI[B]
  abstract class AbstractTransformedS[+B] extends Seq[B] with super[IterableViewLike].TransformedI[B] with TransformedS[B]
}

trait TraversableViewLike[+A, +Coll, +This <: TraversableView[A, Coll] with TraversableViewLike[A, Coll, This]] {

  trait TransformedT[+B] extends TraversableView[B, Coll]
}

trait IndexedSeqView[A, +Coll] extends IndexedSeq[A]
                                with IndexedSeqOptimized[A, IndexedSeqView[A, Coll]]
                                with SeqView[A, Coll]
                                with SeqViewLike[A, Coll, IndexedSeqView[A, Coll]] {

  trait TransformedX[B] extends IndexedSeqView[B, Coll] with super.TransformedS[B]
  abstract class AbstractTransformedX[B] extends super.AbstractTransformedS[B] with TransformedX[B]
}

[smarter]

Minimization:

import scala.collection._

abstract class ATS[B] extends Seq[B]

trait ISV[A, +Coll] extends IndexedSeq[A]
                       with IndexedSeqOptimized[A, ISV[A, Coll]] {
  abstract class Fail[B] extends ATS[B] with ISV[B, Coll]
}

[smarter]

@allanrenucci Your reproduction (and my minimization) both fail to compile using scalac with:

i5433.scala:5: error: overriding method newBuilder in trait TraversableLike of type => scala.collection.mutable.Builder[A,ISV[A,Coll]];
 method newBuilder in trait GenericTraversableTemplate of type => scala.collection.mutable.Builder[A,IndexedSeq[A]] has incompatible type
trait ISV[A, +Coll] extends IndexedSeq[A]
      ^
one error found

So they're not actually valid reproduction of the original problem, but point to a separate issue in Dotty (missing check in RefChecks)

[smarter]

Here's a minimization that:

• Crashes with a Ycheck failure after ResolveSuper with dotty
• Compiles with scalac
• ... but also crashes at runtime with a ClassCastException when compiled with scalac. Indicating that scalac is also miscompiling that code (the crash only happens when we call super.foo):

class X
class Y extends X

trait A[+T] {
  def foo: T = null.asInstanceOf[T]
}

trait B extends A[X] {
  override def foo: X = new X
}

trait C extends A[Y] {
  override def foo: Y = new Y
  super.foo
}

class Fail extends B with C

object Test {
  def main(args: Array[String]): Unit = {
    new Fail // ClassCastException: X cannot be cast to Y
  }
}

@adriaanm is this a known issue ? ^

[smarter]

To elaborate, the problem is that the call to super.foo in C creates an abstract super-accessor in C like this:

def C$$super$foo: Y

which is then implemented in Fail as:

def C$$super$foo: Y = super[B].foo

which does not typecheck since the result type of B#foo is X, not Y.

Looking at the spec for super-calls, I don't think this situation was ever considered.