Binary Break Affecting Macro Annotations Compiled on 3.3

[j-mie6]

Compiler version

3.6.2 (but also applies as far back as 3.5.0, works on 3.4)

Minimized code

Using scala --dependency com.github.j-mie6::parsley-debug::5.0.0-M9

Then, use the annotation @parsley.debuggable, which is a macro annotation compiled on 3.3.1.

@parsley.debuggable object foo

Output (click arrow to expand)

  unhandled exception while running inlining on rs$line$1

  An unhandled exception was thrown in the compiler.
  Please file a crash report here:
  https://github.com/scala/scala3/issues/new/choose
  For non-enriched exceptions, compile with -Xno-enrich-error-messages.

     while compiling: rs$line$1
        during phase: inlining
                mode: Mode(ImplicitsEnabled,ReadPositions,Interactive)
     library version: version 2.13.15
    compiler version: version 3.6.2
            settings: -Xcook-docs true -Xread-docs true -classpath /home/jamie/.cache/coursier/arc/https/github.com/scala/scala3/releases/download/3.6.2/scala3-3.6.2-x86_64-pc-linux.tar.gz/scala3-3.6.2-x86_64-pc-linux/maven2/org/scala-lang/scala3-library_3/3.6.2/scala3-library_3-3.6.2.jar:/home/jamie/.cache/coursier/v1/https/repo1.maven.org/maven2/com/github/j-mie6/parsley-debug_3/5.0.0-M9/parsley-debug_3-5.0.0-M9.jar:/home/jamie/.cache/coursier/arc/https/github.com/scala/scala3/releases/download/3.6.2/scala3-3.6.2-x86_64-pc-linux.tar.gz/scala3-3.6.2-x86_64-pc-linux/maven2/org/scala-lang/scala-library/2.13.15/scala-library-2.13.15.jar:/home/jamie/.cache/coursier/v1/https/repo1.maven.org/maven2/com/github/j-mie6/parsley_3/5.0.0-M9/parsley_3-5.0.0-M9.jar -d <REPL compilation output>

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at scala.reflect.Selectable.applyDynamic(Selectable.scala:40)
        at scala.reflect.Selectable.applyDynamic$(Selectable.scala:11)
        at scala.reflect.Selectable$DefaultSelectable.applyDynamic(Selectable.scala:51)
        at dotty.tools.dotc.transform.MacroAnnotations$.callMacro(MacroAnnotations.scala:113)
        at dotty.tools.dotc.transform.MacroAnnotations$.$anonfun$1(MacroAnnotations.scala:68)
        at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
        at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
        at scala.collection.immutable.List.foldLeft(List.scala:79)
        at dotty.tools.dotc.transform.MacroAnnotations$.expandAnnotations(MacroAnnotations.scala:63)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transformMemberDef(Inlining.scala:115)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:71)
        at dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1265)
        at dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1265)
        at dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1267)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:58)
        at dotty.tools.dotc.ast.TreeMapWithTrackedStats.transform(TreeMapWithTrackedStats.scala:63)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:95)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1605)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
        at dotty.tools.dotc.ast.TreeMapWithTrackedStats.transform(TreeMapWithTrackedStats.scala:60)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transformMemberDef(Inlining.scala:135)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:71)
        at dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1265)
        at dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1265)
        at dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1267)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1616)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
        at dotty.tools.dotc.ast.TreeMapWithTrackedStats.transform(TreeMapWithTrackedStats.scala:48)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:75)
        at dotty.tools.dotc.transform.Inlining$$anon$2.transform(Inlining.scala:57)
        at dotty.tools.dotc.transform.MacroTransform.run(MacroTransform.scala:20)
        at dotty.tools.dotc.transform.Inlining.run(Inlining.scala:39)
        at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:380)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:334)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:373)
        at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:343)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
        at dotty.tools.dotc.Run.runPhases$1(Run.scala:336)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:384)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:396)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:69)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:396)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:288)
        at dotty.tools.repl.ReplCompiler.compile(ReplCompiler.scala:88)
        at dotty.tools.repl.ReplDriver.compile(ReplDriver.scala:334)
        at dotty.tools.repl.ReplDriver.interpret(ReplDriver.scala:296)
        at dotty.tools.repl.ReplDriver.loop$1(ReplDriver.scala:209)
        at dotty.tools.repl.ReplDriver.runUntilQuit$$anonfun$1(ReplDriver.scala:212)
        at dotty.tools.repl.ReplDriver.withRedirectedOutput(ReplDriver.scala:251)
        at dotty.tools.repl.ReplDriver.runBody$$anonfun$1(ReplDriver.scala:225)
        at dotty.tools.runner.ScalaClassLoader$.asContext(ScalaClassLoader.scala:80)
        at dotty.tools.repl.ReplDriver.runBody(ReplDriver.scala:225)
        at dotty.tools.repl.ReplDriver.runUntilQuit(ReplDriver.scala:212)
        at dotty.tools.repl.ReplDriver.tryRunning(ReplDriver.scala:149)
        at dotty.tools.repl.Main$.main(Main.scala:7)
        at dotty.tools.repl.Main.main(Main.scala)
Caused by: java.lang.AbstractMethodError
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        ... 61 more

This also happens outside the REPL. From my understanding of AbstractMethodError, this means the binary API of macro-annotations was broken from 3.3/3.4 at 3.5.

[j-mie6]

A quick check within the parsley codebase suggests:

@experimental class debuggable extends MacroAnnotation {
[error]    |                    ^
[error]    |class debuggable needs to be abstract, since def transform
[error]    |  (using x$1: scala.quoted.Quotes)
[error]    |    (definition: x$1.reflect.Definition, companion:
[error]    |      Option[x$1.reflect.Definition]): List[x$1.reflect.Definition] in trait MacroAnnotation in package scala.annotation is not defined 
[error]    |(The class implements a member with a different type: def transform
[error]    |  (using x$1: scala.quoted.Quotes)
[error]    |    (tree: x$1.reflect.Definition): List[x$1.reflect.Definition] in class debuggable in package parsley)

Which is indeed a binary break. Is it even possible to resolve this? If the API for macro-annotations is broken from the LTS 3.3 series, then basically no cross-compiled libraries would be able to use them!

I would guess this is possible to resolve by making this method non-abstract and have a forwarder to the new method with a default argument provided.

[hamzaremmal]

@j-mie6 MacroAnnotation is an experimental API. It has no guaranteed backward binary or source compatibility. See the documentation about @experimental here.

[j-mie6]

I understand that, however, in this case this is fixable with a simple forwarder method, I believe.

[hamzaremmal]

Sure, but that's not the API we want to have (for now), see SIP-63.

[j-mie6]

As a mitigation for Scala 3.3 targeting library authors, just add

def transform(using Quotes)(tree: quotes.reflect.Definition, companion: Option[quotes.reflect.Definition]): List[quotes.reflect.Definition] = transform(tree)

To your annotation class, it will work again