Support calling signature polymorphic methods (needed to use `MethodHandle` and `VarHandle`)

[smarter]

See scala/scala#4139, scala/scala#5594, scala/scala#9530, scala/scala#9930

[smarter]

Looks like VarHandle.setVolatile requires some extra stuff not supported even by scala 2 currently: scala/bug#12348 (EDIT: fixed in scala/scala#9530)

[ScalaWilliam]

Hi @smarter is there a workaround / hackaround for this by any chance?

Eg some way to generate pure Java at compile-time or similar?

If it works, Scala 3 macros + VarHandles/MethodHandles = <3

[smarter]

I don't think there's any workaround besides writing the calls in java files

[ScalaWilliam]

Thanks for getting back so quickly @smarter .

Just to add, I was doing simple calls like VarHandle#get and MethodHandle.invokeExact just fine in Scala 2.13.6 (compile & runtime), but it fails to compile in Scala 3.0.1, for example:

|    var memoryAddress: MemoryAddress = ref.invokeExact(
[error]     |                                       ^
[error]     |                         Found:    Object
[error]     |                         Required: jdk.incubator.foreign.MemoryAddress

and:

[error] -- [E007] Type Mismatch Error: 
[error] 198 |    val rootVal: MemoryAddress = addrHandle.get(theDoc)
[error]     |                                 ^^^^^^^^^^^^^^^^^^^^^^
[error]     |                         Found:    Object
[error]     |                         Required: jdk.incubator.foreign.MemoryAddress

Changing the code to have type-casting with asInstanceOf compiles, although fails at runtime:

java.lang.invoke.WrongMethodTypeException: expected (MemoryAddress,int,int,MemoryAddress,MemoryAddress)MemoryAddress but found (Object[])Object

I do appreciate it's a completely new compiler - because it looks like a regression (or I missed some sort of migration document), would there be a chance to prioritize this? I imagine there might be a few Scala codebases that might not be able to take the leap because of this.

[smarter]

I'd be happy to help along anyone who wants to port the scala 2 implementation of this feature but I don't have the time to deal with it myself, and I don't know if anyone else does.

[vasilmkd]

I would like to work on this issue, but I have not yet contributed to the Scala 3 compiler and I'm honestly not sure where to begin. It seems to me that the Scala 2 code cannot be easily ported to the new compiler. I'd appreciate any pointers.

[markehammons]

Seconding the above. I looked, and I don't know how the scala 2 code corresponds to the scala 3 code.

[smarter]

Welcome! Some pointers:

• The brand new guide to contributing to the compiler: https://docs.scala-lang.org/scala3/guides/contribution/contribution-intro.html
• The documentation of MethodHandle contains a section "Method handle compilation" which describes how signature polymorphic methods work: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/invoke/MethodHandle.html
• This is also described more formally in the Java Language Specification, see "If the compile-time declaration for the method invocation is a signature polymorphic method, then:" in https://docs.oracle.com/javase/specs/jls/se17/html/jls-15.html#jls-15.12.3
• Scala 2 implements this in the typechecker, when we type the application of a method to its term arguments: https://github.com/scala/scala/blob/0a8b8b98953d1c8a80337515a13036a2dca73785/src/compiler/scala/tools/nsc/typechecker/Typers.scala#L3585-L3602 (in Scala 3 the correspond method is called typedApply and is in Applications.scala). There's also some logic in the backend, but as far as I can tell this is only used when combined with the Scala 2 optimize and @inline which don't exist in Scala 3.
• The original Scala 2 PR discusses why this specific implementation was chosen (SI-7965 Support calls to MethodHandle.{invoke,invokeExact} scala#4139) and is worth reading in full.

I think the Scala 2 approach could work in Scala 3 too, but we also store our typed trees into tasty and can deserialize them at any point (for example, when calling an inline def), so the deserializer would likely need to use the same symbol creation logic (https://github.com/lampepfl/dotty/blob/6e680453fc24cc9e9d2bb873638d76fec3883375/compiler/src/dotty/tools/dotc/core/tasty/TreeUnpickler.scala#L1138-L1140).

---

It might be worth exploring alternative approaches which don't involve creating fake symbols, but as mentioned in scala/scala#4139 this is likely to touch many parts of the compiler. I had a quick look just now and it seems pretty daunting indeed.

[vasilmkd]

Thank you for the great list of resources @smarter. I will take some time to internalize everything and hopefully come up with a PR. Thank you again.