Manual dotty bootstrap fails

[odersky]

I just did this:

/Users/odersky/workspace/dotty/src/dotty/tools/dotc> time java -Xms1g -Xmx3g dotty.tools.dotc.Main -d /classes *.scala */*.scala */*/*.scala

And then did it again (with /classes at the head of the classpath)
I got at first:

Exception in thread "main" java.lang.NoClassDefFoundError: Ldotty/tools/dotc/core/Phases$PhasesBase$TerminalPhase$;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2575)
at java.lang.Class.getDeclaredField(Class.java:2060)
at dotty.runtime.LazyVals$.getOffset(LazyVals.scala:71)

This pointed to a problem that somewhere a reference to a removed empty companion object survived. I disabled the dropEmptyCompanion phase and then that part worked. (I instrumented the "getOffset" method and got this.

/Users/odersky/workspace/dotty/src/dotty/tools/dotc> time java -Xms1g -Xmx3g dotty.tools.dotc.Main -d /classes *.scala */*.scala */*/*.scala
getting offset of class dotty.tools.dotc.core.Contexts$ContextBase . bitmap$0
getting offset of class dotty.tools.dotc.config.PathResolver . bitmap$0
getting offset of class dotty.tools.dotc.core.Contexts$Context . bitmap$0
getting offset of class dotty.tools.dotc.core.unpickleScala2.Scala2Unpickler . bitmap$0
getting offset of class dotty.tools.dotc.core.StdNames$ScalaNames . bitmap$0
getting offset of class dotty.tools.dotc.core.classfile.ClassfileParser . bitmap$0
getting offset of class dotty.tools.dotc.transform.PatternMatcher$Translator . bitmap$0
getting offset of class dotty.tools.dotc.transform.PatternMatcher$Translator . bitmap$0
getting offset of class dotty.tools.dotc.core.Definitions . bitmap$0
getting offset of class dotty.tools.dotc.core.StdNames$ScalaTermNames . bitmap$0
getting offset of class dotty.tools.dotc.typer.VarianceChecker . bitmap$0

So, this is fine. But after some more time compiling with the bootstrapped compiler, I got this:

Exception in thread "main" java.lang.ClassCastException: dotty.tools.dotc.core.Types$CachedThisType cannot be cast to dotty.tools.dotc.core.Types$TypeRef
at dotty.tools.dotc.core.TypeApplications$.$anonfun$appliedTo$extension3$1(TypeApplications.scala:484)
at dotty.tools.dotc.core.TypeApplications$$$Lambda$405/2114444063.apply(Unknown Source)

This raises the following questions:

1. Do we really use the bootstrapped dotty compiler in partest-bootstrapped? It seems this is impossible, given the failures I have observed.
2. Eliminating empty companion objects is very important: generated code went up from ~30K blocks to ~34K blocks, a 13.3% increase, after dropEmptyCompanions was disabled. But it seems very hard to do in a miniphase (not sure about the backend). Concretely in the case at hand I believe an accessor got copied into the inheriting class Context. When eliminating the companion object of the base class Phases #TerminalPhase, that accessor was overlooked. Is it even possible to remove all accessors in separate compilation?
3. Where does the classcast exception come from?

[odersky]

Checking the compiler output confirmed my suspicion. When compiling Contexts.scala and Phases.scala together:

dotc core/Phases.scala core/Contexts.scala -Xprint:restore -d /classes

we get this after phase restoreScope:

    val TerminalPhase$lzy: PhasesBase.this.TerminalPhase$
    final lazy <accessor> module def TerminalPhase(): 
      PhasesBase.this.TerminalPhase$ = this.TerminalPhase$lzywe get this in Contexts.scala

So, it's a reference to the non-existing object PhasesBase.this.TerminalPhase$ in the inheriting class Context.

[DarkDimius]

generated code went up from went up from ~30K blocks to ~34K blocks,

could you please allaborate how this number was measured?
If you have measured number of blocks in program, than given data from #510 I do not think this makes a difference(0.13*0.05= 0.0065)

Do we really use the bootstrapped dotty compiler in partest-bootstrapped?

I'm investigating

[DarkDimius]

This pointed to a problem that somewhere a reference to a removed empty companion

If lazy vals was looking for a field in the companion object, I would expect that companion object was not empty, otherwise there would be not need to reference it.

Do we really use the bootstrapped dotty compiler in partest-bootstrapped?

No, we're not. See #1115 And we have blocking bugs that stop us from doing it.

Where does the classcast exception come from?

Looks similar to #840.

[odersky]

I don't think the class cast exception is similar to #840. Some more observations. The failing line reads:

 case EtaExpansion(self1) =>
    self1.appliedTo(args)

EtaExpansion is an extractor, with an unapply method that looks as follows:

def unapply(tp: Type)(implicit ctx: Context): Option[TypeRef] = {
  def argsAreForwarders(args: List[Type], n: Int): Boolean = args match {
    case Nil =>
      n == 0
    case TypeRef(RefinedThis(rt), sel) :: args1 =>
      rt.eq(tp) && sel == tpnme.hkArg(n - 1) && argsAreForwarders(args1, n - 1)
    case _ =>
      false
  }
  tp match {
    case TypeLambda(_, argBounds, AppliedType(fn: TypeRef, args))
    if argsAreForwarders(args, tp.typeParams.length) => Some(fn)
    case _ => None
  }
}

I'll try to dig deeper on this.

[DarkDimius]

fix for the companion problem proposed in #1117

[DarkDimius]

I'm also diving into fixing ClassCastException now. Please tell me if you're already working on a fix.

[odersky]

I got some more info on the ClassCastException problem. It seems dotc output is different depending whether one file is compiled or the whole compiler is compiled with it. The failing code is in TypeApplications, line 483-484:

else self/*5*/.stripTypeVar match {
  case EtaExpansion(self1) =>
    self1.appliedTo(args)

If we compile TypeApplications alone, we get this code:

public final dotty.tools.dotc.core.Types$Type appliedTo$extension3(dotty.tools.dotc.core.Types$Type, scala.collection.immutable.List, dotty.tools.dotc.core.Contexts$Context);
  Code:
     0: goto          282
     3: aload         6
     5: invokevirtual #321                // Method scala/collection/immutable/List.isEmpty:()Z
     8: ifne          25
    11: goto          14
    14: aload         7
    16: invokevirtual #811                // Method dotty/tools/dotc/core/Contexts$Context.erasedTypes:()Z
    19: ifne          25
    22: goto          30
    25: aload         5
    27: goto          279
    30: aload         5
    32: aload         7
    34: invokevirtual #631                // Method dotty/tools/dotc/core/Types$Type.stripTypeVar:(Ldotty/tools/dotc/core/Contexts$Context;)Ldotty/tools/dotc/core/Types$Type;
    37: astore        8
    39: goto          276
    42: getstatic     #797                // Field dotty/tools/dotc/core/TypeApplications$EtaExpansion$.MODULE$:Ldotty/tools/dotc/core/TypeApplications$EtaExpansion$;
    45: aload         8
    47: aload         7
    49: invokevirtual #814                // Method dotty/tools/dotc/core/TypeApplications$EtaExpansion$.unapply:(Ldotty/tools/dotc/core/Types$Type;Ldotty/tools/dotc/core/Contexts$Context;)Lscala/Option;
    52: astore        9
    54: aload         9
    56: invokevirtual #643                // Method scala/Option.isDefined:()Z
    59: ifne          65
    62: goto          95
    65: aload         9
    67: invokevirtual #647                // Method scala/Option.get:()Ljava/lang/Object;
    70: checkcast     #193                // class dotty/tools/dotc/core/Types$TypeRef
    73: astore        10
    75: aload         10
    77: astore        11
    79: getstatic     #553                // Field dotty/tools/dotc/core/Types$.MODULE$:Ldotty/tools/dotc/core/Types$;
    82: aload         11
    84: invokevirtual #560                // Method dotty/tools/dotc/core/Types$.decorateTypeApplications:(Ldotty/tools/dotc/core/Types$Type;)Ldotty/tools/dotc/core/Types$Type;
    87: astore        5
    89: goto          3
    92: nop
    93: nop
    94: athrow

This code looks good. But we we compile the whole compiler, we get this code:

public final dotty.tools.dotc.core.Types$Type appliedTo$extension3(dotty.tools.dotc.core.Types$Type, scala.collection.immutable.List, dotty.tools.dotc.core.Contexts$Context);
  Code:
     0: goto          288
     3: aload         6
     5: invokevirtual #321                // Method scala/collection/immutable/List.isEmpty:()Z
     8: ifne          25
    11: goto          14
    14: aload         7
    16: invokevirtual #806                // Method dotty/tools/dotc/core/Contexts$Context.erasedTypes:()Z
    19: ifne          25
    22: goto          30
    25: aload         5
    27: goto          285
    30: aload         5
    32: aload         7
    34: invokevirtual #631                // Method dotty/tools/dotc/core/Types$Type.stripTypeVar:(Ldotty/tools/dotc/core/Contexts$Context;)Ldotty/tools/dotc/core/Types$Type;
    37: astore        8
    39: goto          282
    42: getstatic     #792                // Field dotty/tools/dotc/core/TypeApplications$EtaExpansion$.MODULE$:Ldotty/tools/dotc/core/TypeApplications$EtaExpansion$;
    45: aload         8
    47: aload         7
    49: invokevirtual #809                // Method dotty/tools/dotc/core/TypeApplications$EtaExpansion$.unapply:(Ldotty/tools/dotc/core/Types$Type;Ldotty/tools/dotc/core/Contexts$Context;)Lscala/Option;
    52: astore        9
    54: aload         9
    56: invokevirtual #814                // Method scala/Option.isDefined:()Z
    59: ifne          65
    62: goto          101
    65: aload         9
    67: invokevirtual #818                // Method scala/Option.get:()Ljava/lang/Object;
    70: checkcast     #193                // class dotty/tools/dotc/core/Types$TypeRef
    73: astore        10
    75: aload         10
    77: invokevirtual #677                // Method dotty/tools/dotc/core/Types$TypeRef._1:()Ldotty/tools/dotc/core/Types$Type;
    80: checkcast     #193                // class dotty/tools/dotc/core/Types$TypeRef
    83: astore        11
    85: getstatic     #553                // Field dotty/tools/dotc/core/Types$.MODULE$:Ldotty/tools/dotc/core/Types$;
    88: aload         11
    90: invokevirtual #560                // Method dotty/tools/dotc/core/Types$.decorateTypeApplications:(Ldotty/tools/dotc/core/Types$Type;)Ldotty/tools/dotc/core/Types$Type;
    93: astore        5
    95: goto          3
    98: nop
    99: nop
   100: athrow

The difference is at position 77. In the good code we directly copy the result of the extractor into the variable self1 (which is of type TypeRef). In the bad code, we copy the first element of the extractor result, which is wrong.

The tree after pattern matching is this. I annotated the point where the difference occurs.

          def case621(): dotty.tools.dotc.core.Types.Type = {
            val o75: Option = 
              dotty.tools.dotc.core.TypeApplications.EtaExpansion.unapply(
                selector125
              , ctx)
            if o75.isDefined() then {
              val x215: dotty.tools.dotc.core.Types.TypeRef = 
                o75.get().asInstanceOf[dotty.tools.dotc.core.Types.TypeRef]
              {
                val self1: dotty.tools.dotc.core.Types.TypeRef = x215  <<<<<< That's where the difference happens
                tailLabel100($this, 
                  dotty.tools.dotc.core.Types.decorateTypeApplications(self1
                    )
                , args, ctx)
              }
            } else {

[odersky]

I just added a lot more info to the original issue.

On Sat, Feb 20, 2016 at 3:53 PM, Dmitry Petrashko notifications@github.com
wrote:

I'm also diving into fixing ClassCastException now. Please tell me if
you're already working on a fix.

—
Reply to this email directly or view it on GitHub
#1114 (comment).

Martin Odersky
EPFL

[odersky]

More info: When compiling the whole compiler, after pattern matching, this code is generated instead of the one shown above:

             val o74: Option[dotty.tools.dotc.core.Types.TypeRef] = 
                dotty.tools.dotc.core.TypeApplications.EtaExpansion.unapply(
                  selector1499
                )(ctx)
              if o74.isDefined then {
                val x2256: dotty.tools.dotc.core.Types.TypeRef = o74.get
                {
                  val self1: dotty.tools.dotc.core.Types.TypeRef =   <<<<<<<<< here's where it goes wrong
                    x2256._1.asInstanceOf[
                      dotty.tools.dotc.core.Types.TypeRef
                    ]
                  {
                    {
                      tailLabel2080($this)(
                        dotty.tools.dotc.core.Types.decorateTypeApplications
                          (
                        self1)
                      )(args)(ctx)
                    }
                  }
                }
              } else case15041()

So it looks like the PatternMatcher introduces the incorrect code, but only if the file is compiled with other files in the compiler.

[odersky]

That's all I have to offer by way of diagnosis.