java.lang.NullPointerException on pattern match in dotty REPL

[prayagupa]

Might be more of stackoverflow question not a issues at all but I was playing with dotty REPL with a tuple2 and _._2 works fine

scala> Seq(("Porcupine", 1), ("Floyd", 2), ("Phil Collins", 3)).map(_._2) 
val res0: Seq[Int] = List(1, 2, 3)

But when I pattern match and get the second column it throws a NullPointerException.

scala> Seq(("Porcupine", 1), ("Floyd", 2), ("Phil Collins", 3)).collect {case (name, value) => value } 
Exception in thread "main" java.lang.NullPointerException
	at dotty.tools.dotc.transform.ExpandPrivate.ensurePrivateAccessible(ExpandPrivate.scala:93)
	at dotty.tools.dotc.transform.ExpandPrivate.transformSelect(ExpandPrivate.scala:104)
	at dotty.tools.dotc.transform.MegaPhase.goSelect(MegaPhase.scala:531)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:212)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:364)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:248)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:365)
	at dotty.tools.dotc.transform.MegaPhase.mapDefDef$1(MegaPhase.scala:228)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:231)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:364)
	at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:373)
	at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:378)
	at scala.collection.immutable.List.mapConserve(List.scala:176)
	at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:378)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:317)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:365)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:234)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:364)
	at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:373)
	at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:378)
	at scala.collection.immutable.List.mapConserve(List.scala:176)
	at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:378)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:317)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:365)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:234)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:364)
	at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:373)
	at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:378)
	at scala.collection.immutable.List.mapConserve(List.scala:176)
	at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:378)
	at dotty.tools.dotc.transform.MegaPhase.mapPackage$1(MegaPhase.scala:334)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:337)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:365)
	at dotty.tools.dotc.transform.MegaPhase.transformUnit(MegaPhase.scala:384)
	at dotty.tools.dotc.transform.MegaPhase.run(MegaPhase.scala:396)
	at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:293)
	at scala.collection.immutable.List.map(List.scala:283)
	at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:295)
	at dotty.tools.dotc.Run.runPhases$4$$anonfun$4(Run.scala:123)
	at scala.compat.java8.JProcedure1.apply(JProcedure1.java:18)
	at scala.compat.java8.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:191)
	at dotty.tools.dotc.Run.runPhases$5(Run.scala:136)
	at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:141)
	at scala.compat.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
	at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:88)
	at dotty.tools.dotc.Run.compileUnits(Run.scala:143)
	at dotty.tools.dotc.Run.compileUnits(Run.scala:99)
	at dotty.tools.repl.ReplCompiler.runCompilationUnit(ReplCompiler.scala:188)
	at dotty.tools.repl.ReplCompiler.compile(ReplCompiler.scala:197)
	at dotty.tools.repl.ReplDriver.compile(ReplDriver.scala:223)
	at dotty.tools.repl.ReplDriver.interpret(ReplDriver.scala:196)
	at dotty.tools.repl.ReplDriver.runUntilQuit(ReplDriver.scala:149)
	at dotty.tools.repl.Main$.main(Main.scala:6)
	at dotty.tools.repl.Main.main(Main.scala)

Get the same error with Seq(("Porcupine", 1), ("Floyd", 2), ("Phil Collins", 3)).collect {case x => x._2}.

Tried a bit exploring the reason which is in ExpandPrivate#ensurePrivateAccessible which is line assert(d.symbol.sourceFile != null && isSimilar(d.symbol.sourceFile.path, ctx.owner.sourceFile.path)

  private def ensurePrivateAccessible(d: SymDenotation)(implicit ctx: Context) =
    if (isVCPrivateParamAccessor(d))
      d.ensureNotPrivate.installAfter(thisPhase)
    else if (d.is(PrivateTerm) && !d.owner.is(Package) && d.owner != ctx.owner.lexicallyEnclosingClass) {
      // Paths `p1` and `p2` are similar if they have a common suffix that follows
      // possibly different directory paths. That is, their common suffix extends
      // in both cases either to the start of the path or to a file separator character.
      def isSimilar(p1: String, p2: String): Boolean = {
        var i = p1.length - 1
        var j = p2.length - 1
        while (i >= 0 && j >= 0 && p1(i) == p2(j) && p1(i) != separatorChar) {
          i -= 1
          j -= 1
        }
        (i < 0 || p1(i) == separatorChar) &&
        (j < 0 || p1(j) == separatorChar)
      }

      assert(d.symbol.sourceFile != null &&
             isSimilar(d.symbol.sourceFile.path, ctx.owner.sourceFile.path),
          s"private ${d.symbol.showLocated} in ${d.symbol.sourceFile} accessed from ${ctx.owner.showLocated} in ${ctx.owner.sourceFile}")
      d.ensureNotPrivate.installAfter(thisPhase)
    }

I also tried with simpler pattern match, it does not work.

Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 10).collect {case x if x % 2 == 0 => (x, 1) }

Don't know what is SymDenotation.symbol or Context.owner. I installed dotty using homebrew and believe it is the built from latest codebase.

[Blaisorblade]

Thanks for the report. That’s a crash in the compiler, those are always bugs. Dotty from homebrew is likely the latest release. Todo: recheck on latest master.

[allanrenucci]

Can be reproduced with List(1, 2, 3).collect { case x => x }. ctx.owner.sourceFile is null in the REPL: https://github.com/lampepfl/dotty/blob/8bd162554d266a23896eb0833a264b57873d3140/compiler/src/dotty/tools/dotc/transform/ExpandPrivate.scala#L93

[expz]

There is no error with an explicit PartialFunction:

scala> val id = new PartialFunction[Int, Int] {
         def apply(x: Int) = x
         def isDefinedAt(x: Int) = true
       } 
val id: PartialFunction[Int, Int] = <function1>
scala> Seq(1, 2, 3).collect(id) 
val res1: Seq[Int] = List(1, 2, 3)

But even an identity function fails:

scala> def id(x: Int) = x 
def id(x: Int): Int
scala> Seq(1, 2, 3).collect(id) 
Exception in thread "main" java.lang.NullPointerException
	at dotty.tools.dotc.transform.ExpandPrivate.ensurePrivateAccessible(ExpandPrivate.scala:93)
        ...

[Blaisorblade]

@allanrenucci is the sourcefile supposed to be null? If so, can we relax the assertion? It seems some of its checks should become proper conditions rather than an assert?

[allanrenucci]

By quickly looking at it, I would say sourcefile should not be null. When running within the REPL, the source file is a dotty.tools.io.VirtualFile. I am not sure why null is returned instead

[expz]

Here is the reason null is returned. An anonymous class rs$line$2$$anon$1 is defined and then instantiated as an object of type PartialFunction. The sourceFile of the anonymous class is null. This happens, because:

1. sourceFile first tries to read the assocFile of the ClassSymbol and finds that it is null.

2. So it tries to create the source file from the myAnnotations of (the denotation of) the ClassSymbol. Unfortunately, all the information the annotation (a SourceFile(Apply(...))) contains is the filename as a string. So sourceFile tries to recreate an AbstractFile by calling AbstractFile.getFile. In the case of the normal compiler, this works and returns the PlainFile of the scala source file. But in the case of the REPL, it tries to find rs$line$2 in the filesystem and fails, returning null.

I see two options. A hack to fix this is to just create a VirtualFile if the PlainFile cannot be created:

    final def sourceFile(implicit ctx: Context): AbstractFile = {
      val file = associatedFile
      if (file != null && !file.path.endsWith("class")) file
      else {
        val topLevelCls = denot.topLevelClass(ctx.withPhaseNoLater(ctx.flattenPhase))
        topLevelCls.getAnnotation(defn.SourceFileAnnot) match {
          case Some(sourceAnnot) => sourceAnnot.argumentConstant(0) match {
            case Some(Constant(path: String)) => // Changes here:
              val plainFile = AbstractFile.getFile(path)
              if (plainFile != null) plainFile
              else new dotty.tools.io.VirtualFile(path)
            case none => null
          }
          case none => null
        }
      }
    }

This solves the error and produces no errors in dotty-compiler/test or testCompilation. But it looks ugly to me. It introduces a dependency on VirtualFile where it seems out of place. And perhaps this code in dotty.tools.dotc.interactive.SourceTree relies on returning null for a non-existant path in the SourceFile(...) annotation?

object SourceTree {
  def fromSymbol(sym: ClassSymbol, id: String = "")(implicit ctx: Context): Option[SourceTree] = {
    if (sym == defn.SourceFileAnnot || // FIXME: No SourceFile annotation on SourceFile itself
        sym.sourceFile == null) // FIXME: We cannot deal with external projects yet
      None
    ...

The second option would be to decide that the anonymous class which becomes a PartialFunction object should have assocFile != null. But I do not know whether that matches the intended meaning of assocFile.

[allanrenucci]

I don't think associatedFile should return null. Whether the source is a normal file or a virtual one shouldn't matter.