[native-image] runs forever on closure compiler

[cushon]

native-image (built at a6781fa) appears to hang when processing https://github.com/google/closure-compiler. The following invocation ran for >90 minutes without finishing:

$ wget https://dl.google.com/closure-compiler/compiler-latest.zip
$ unzip closure-compiler-v20180506.jar
./latest_graalvm/graalvm-1.0.0-rc2-dev/bin/native-image -jar closure-compiler-v20180506.jar
Build on Server(pid: 28200, port: 35863)*
   classlist:     987.20 ms
       (cap):     869.28 ms
       setup:   1,583.04 ms
RecomputeFieldValue.ArrayBaseOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayBaseOffset(Class) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Add a RecomputeFieldValue.ArrayBaseOffset manual substitution for com.google.protobuf.UnsafeUtil.
RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Add a RecomputeFieldValue.FieldOffset manual substitution for com.google.protobuf.UnsafeUtil.
...

[concavelenz]

My run did eventually stop with an OutOfMemoryError but I failed to time it:

Build on Server(pid: 247414, port: 26681)*
classlist: 1,010.54 ms
(cap): 1,168.40 ms
setup: 2,064.06 ms
RecomputeFieldValue.ArrayBaseOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayBaseOffset(Class) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Add a RecomputeFieldValue.ArrayBaseOffset manual substitution for com.google.protobuf.UnsafeUtil.
RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Add a RecomputeFieldValue.FieldOffset manual substitution for com.google.protobuf.UnsafeUtil.

analysis: 7,598,531.46 ms

fatal error: java.lang.OutOfMemoryError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:398)
at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:240)
at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:337)
at com.oracle.svm.hosted.server.NativeImageBuildServer.executeCompilation(NativeImageBuildServer.java:378)
at com.oracle.svm.hosted.server.NativeImageBuildServer.lambda$processCommand$8(NativeImageBuildServer.java:315)
at com.oracle.svm.hosted.server.NativeImageBuildServer.withJVMContext(NativeImageBuildServer.java:396)
at com.oracle.svm.hosted.server.NativeImageBuildServer.processCommand(NativeImageBuildServer.java:312)
at com.oracle.svm.hosted.server.NativeImageBuildServer.processRequest(NativeImageBuildServer.java:256)
at com.oracle.svm.hosted.server.NativeImageBuildServer.lambda$serve$7(NativeImageBuildServer.java:216)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
Error: Processing image build request failed

[cstancu]

This is probably the same issue as reported in #411. See also my response there: #411 (comment). I will investigate it soon.

[cstancu]

The problem reported above is fixed. You can get the fixes by building from source. After running mx build, assuming that your working directory is substratevm and that you have closure-compiler-v20180506.jar there, you can run:

$ mx native-image -H:IncludeResourceBundles=com.google.javascript.rhino.Messages,org.kohsuke.args4j.Messages,com.google.javascript.jscomp.parsing.ParserConfig -H:IncludeResources="(externs.zip)|(com/google/javascript/jscomp/js/.*)" -H:+ReportUnsupportedElementsAtRuntime -jar closure-compiler-v20180506.jar
   classlist:   1,693.21 ms
       (cap):   1,046.49 ms
       setup:   2,069.54 ms
RecomputeFieldValue.ArrayBaseOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayBaseOffset(Class) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Add a RecomputeFieldValue.ArrayBaseOffset manual substitution for com.google.protobuf.UnsafeUtil. 
RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Add a RecomputeFieldValue.FieldOffset manual substitution for com.google.protobuf.UnsafeUtil. 
  (typeflow):  12,276.08 ms
   (objects):   5,992.24 ms
  (features):     145.47 ms
    analysis:  18,979.93 ms
    universe:     848.18 ms
     (parse):   2,290.79 ms
    (inline):   2,758.49 ms
   (compile):  10,922.05 ms
     compile:  17,042.94 ms
       image:   1,817.09 ms
       write:     403.64 ms
     [total]:  43,137.84 ms

This will build the closure-compiler-v20180506 executable in the current directory, which you can run with:

$ ./closure-compiler-v20180506 
The compiler is waiting for input via stdin.
var x = 1 + 1;
var x=2;

To address the automatic substitution failed warnings reported during the image building please refer to the article we published recently, the 'Unsafe memory access' section. I only tried it on the small test above, and the unsafe memory access didn't cause an issue.

[concavelenz]

Thank you!

…

On Fri, Jun 1, 2018, 8:39 PM Codrut Stancu ***@***.***> wrote: The problem reported above is fixed. You can get the fixes by building from source <https://github.com/oracle/graal/tree/master/substratevm#quick-start>. After running mx build, assuming that your working directory is substratevm and that you have closure-compiler-v20180506.jar there, you can run: $ mx native-image -H:IncludeResourceBundles=com.google.javascript.rhino.Messages,org.kohsuke.args4j.Messages,com.google.javascript.jscomp.parsing.ParserConfig -H:IncludeResources="(externs.zip)|(com/google/javascript/jscomp/js/.*)" -H:+ReportUnsupportedElementsAtRuntime -jar closure-compiler-v20180506.jar classlist: 1,693.21 ms (cap): 1,046.49 ms setup: 2,069.54 ms RecomputeFieldValue.ArrayBaseOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayBaseOffset(Class) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Add a RecomputeFieldValue.ArrayBaseOffset manual substitution for com.google.protobuf.UnsafeUtil. RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Add a RecomputeFieldValue.FieldOffset manual substitution for com.google.protobuf.UnsafeUtil. (typeflow): 12,276.08 ms (objects): 5,992.24 ms (features): 145.47 ms analysis: 18,979.93 ms universe: 848.18 ms (parse): 2,290.79 ms (inline): 2,758.49 ms (compile): 10,922.05 ms compile: 17,042.94 ms image: 1,817.09 ms write: 403.64 ms [total]: 43,137.84 ms This will build the closure-compiler-v20180506 executable in the current directory, which you can run with: $ ./closure-compiler-v20180506 The compiler is waiting for input via stdin. var x = 1 + 1; var x=2; To address the automatic substitution failed warnings reported during the image building please refer to the article <https://medium.com/graalvm/instant-netty-startup-using-graalvm-native-image-generation-ed6f14ff7692> we published recently, the 'Unsafe memory access' section. I only tried it on the small test above, and the unsafe memory access didn't cause an issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#428 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABMDKjNJ8gqf7VjUQFd_KN6XCtz2kLpeks5t4gkAgaJpZM4UIA0A> .

[ChadKillingsworth]

For the open source command line version, you also need a reflection config: https://gist.github.com/ChadKillingsworth/137e8064a9933379726b4d23254a3214

The full build command is:

mx native-image \
  -H:IncludeResourceBundles=com.google.javascript.rhino.Messages,org.kohsuke.args4j.Messages,org.kohsuke.args4j.spi.Messages,com.google.javascript.jscomp.parsing.ParserConfig \
  -H:IncludeResources="(externs.zip)|(com/google/javascript/jscomp/js/.*)" \
  -H:+ReportUnsupportedElementsAtRuntime \
  -H:ReflectionConfigurationFiles=graal-reflection-config.json \
  -H:+JNI \
  -jar closure-compiler-1.0-SNAPSHOT.jar