Issue with Java 16

[smoell]

I've noticed an issue with Java 16

time="2021-07-26T07:23:27.015" level=info msg="exec 'java' (cwd=/, handler=com.amazon.LambdaExample)"
time="2021-07-26T07:23:31.193" level=info msg="extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory"
time="2021-07-26T07:23:31.193" level=warning msg="Cannot list external agents" error="open /opt/extensions: no such file or directory"
START RequestId: 62765859-8b68-4bf4-b088-1dddcac0b1d1 Version: $LATEST
Exception in thread "main" java.lang.Error: java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module java.base does not "opens java.util" to unnamed module @270fcfcc
        at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:195)
        at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.main(AWSLambda.java:188)
Caused by: java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module java.base does not "opens java.util" to unnamed module @270fcfcc
        at com.amazonaws.services.lambda.runtime.api.client.util.EnvWriter.<init>(EnvWriter.java:26)
        at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:210)
        at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:193)
        ... 1 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module java.base does not "opens java.util" to unnamed module @270fcfcc
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:177)
        at java.base/java.lang.reflect.Field.setAccessible(Field.java:171)
        at com.amazonaws.services.lambda.runtime.api.client.util.EnvWriter.<init>(EnvWriter.java:21)
        ... 3 more
time="2021-07-26T07:23:31.291" level=warning msg="First fatal error stored in appctx: Runtime.ExitError"
time="2021-07-26T07:23:31.292" level=warning msg="Process 14(java) exited: Runtime exited with error: exit status 1"
time="2021-07-26T07:23:31.292" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2021-07-26T07:23:31.292" level=warning msg="Reset initiated: ReserveFail"
time="2021-07-26T07:23:31.292" level=warning msg="Cannot list external agents" error="open /opt/extensions: no such file or directory"
Exception in thread "main" java.lang.Error: java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module java.base does not "opens java.util" to unnamed module @270fcfcc
        at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:195)
        at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.main(AWSLambda.java:188)
Caused by: java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module java.base does not "opens java.util" to unnamed module @270fcfcc
        at com.amazonaws.services.lambda.runtime.api.client.util.EnvWriter.<init>(EnvWriter.java:26)
        at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:210)
        at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:193)
        ... 1 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module java.base does not "opens java.util" to unnamed module @270fcfcc
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:177)
        at java.base/java.lang.reflect.Field.setAccessible(Field.java:171)
        at com.amazonaws.services.lambda.runtime.api.client.util.EnvWriter.<init>(EnvWriter.java:21)
        ... 3 more
time="2021-07-26T07:23:31.376" level=warning msg="First fatal error stored in appctx: Runtime.ExitError"
time="2021-07-26T07:23:31.376" level=warning msg="Process 33(java) exited: Runtime exited with error: exit status 1"
END RequestId: 0b3081de-2ad5-4e90-8688-03cf43d6fedc
REPORT RequestId: 0b3081de-2ad5-4e90-8688-03cf43d6fedc  Init Duration: 0.38 ms  Duration: 182.94 ms     Billed Duration: 183 ms      Memory Size: 3008 MB    Max Memory Used: 3008 MB

The Dockerfile is

FROM openjdk:16.0.2 as builder

RUN microdnf update \        
    && microdnf -y upgrade
    
RUN microdnf install wget -y 

RUN wget https://mirror.softaculous.com/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz -P /tmp
RUN tar xf /tmp/apache-maven-3.6.3-bin.tar.gz -C /opt
RUN ln -s /opt/apache-maven-3.6.3 /opt/maven

ENV M2_HOME=/opt/maven
ENV MAVEN_HOME=/opt/maven
ENV PATH=${M2_HOME}/bin:${PATH}

COPY ./pom.xml ./pom.xml
COPY src ./src/

RUN mvn -Dmaven.test.skip=true clean package

FROM openjdk:16-slim

RUN apt-get update && apt-get upgrade -y
    
COPY --from=builder target/dependency/* /function/
COPY --from=builder target/LambdaExample.jar /function
    
ENTRYPOINT [ "java", "-cp", "/function/*", "com.amazonaws.services.lambda.runtime.api.client.AWSLambda" ]

CMD ["com.amazon.LambdaExample::handleRequest"]

With JDK 15 as runtime, everything works as expected.

[Raniz85]

I had the same issue and solved it by forking and releasing an internal version.

The issue is that EnvWriter tries to makeCollections.unmodifiableMap modifiable by changing internal fields, which isn't allowed anymore in Java 16.

I solved it by removing all modifications to the environment (by removing all usages of EnvWriter), which seems to work fine. The only thing it does is set some legacy environment variables and set variables for AWS X-Ray tracing.

The X-Ray tracing will probably break because of this but we're not using it so that's not an issue for us.

[Raniz85]

I found another solution to this issue.

You can pass --add-opens java.base/java.util=ALL-UNNAMED to your JVM arguments and the runtime client will be allowed to modify the environment.

This should really be fixed though. The library uses a lot of reflection and that is causing some issues when compiling with Graal native-image.

[vpatil1311]

I found another solution to this issue.

You can pass --add-opens java.base/java.util=ALL-UNNAMED to your JVM arguments and the runtime client will be allowed to modify the environment.

This should really be fixed though. The library uses a lot of reflection and that is causing some issues when compiling with Graal native-image.

Hi .. can you please confirm which Java version you could fix this issue ? I have same issue but the stack trace is different. I am using JDK 17, Junit 5.

org.opentest4j.MultipleFailuresError: Multiple Failures (2 failures)
java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module java.base does not "opens java.util" to unnamed module @1c3a4799
java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module java.base does not "opens java.util" to unnamed module @1c3a4799
at org.junit.vintage.engine.execution.TestRun.getStoredResultOrSuccessful(TestRun.java:196)
at org.junit.vintage.engine.execution.RunListenerAdapter.fireExecutionFinished(RunListenerAdapter.java:226)
at org.junit.vintage.engine.execution.RunListenerAdapter.testFinished(RunListenerAdapter.java:192)
at org.junit.vintage.engine.execution.RunListenerAdapter.testFinished(RunListenerAdapter.java:79)
at org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87)
at org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225)
at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
at org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222)
at org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.mockito.internal.runners.DefaultInternalRunner$1.run(DefaultInternalRunner.java:99)
at org.mockito.internal.runners.DefaultInternalRunner.run(DefaultInternalRunner.java:105)
at org.mockito.internal.runners.StrictRunner.run(StrictRunner.java:40)
at org.mockito.junit.MockitoJUnitRunner.run(MockitoJUnitRunner.java:163)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:84)
at org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:98)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:40)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:529)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:756)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:452)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)

[Raniz85]

@vpatil1311 I was using OpenJDK 16 at the time. I'm no longer involved with that project so can't comment on whether or not they have moved on to later releases.

[vpatil1311]

@smoell Can you please provide details on what fixed the issue for you ? is it to downgrade to JDK 15 ?

[Raniz85]

There's two different solutions in my comments right above your

[vpatil1311]

@Raniz85 thanks for responding. --add-opens approach did work for me when I execute the independent test class inside eclipse. But I am not able to fix my apache MAVEN v3.8 build which uses a maven-surefire-report-plugin to execute the tests. This is a spring boot service. I am using Oracle JDK 17.

[msailes]

@vpatil1311 how are you using Java 17 on Lambda?

[Raniz85]

@vpatil1311 Can't help you there unfortunately. I haven't worked with Maven or Eclipse for almost 10 years.

@msailes Using custom runtimes

[msailes]

@vpatil1311 I imagine there is a property in the surefire plugin to accept additional arguments.

[smirnoal]

I had the same issue and solved it by forking and releasing an internal version.

The issue is that EnvWriter tries to makeCollections.unmodifiableMap modifiable by changing internal fields, which isn't allowed anymore in Java 16.

I solved it by removing all modifications to the environment (by removing all usages of EnvWriter), which seems to work fine. The only thing it does is set some legacy environment variables and set variables for AWS X-Ray tracing.

The X-Ray tracing will probably break because of this but we're not using it so that's not an issue for us.

good stuff, do you have a PR for it?

[vpatil1311]

Just wanted to update the thread for later reference. I could fix it in Maven build as well by using --add-opens in maven-surefire-plugin. maven-surefire-report-plugin is not honoring --add-opens for some reason. Thanks everybody for responding.

[smirnoal]

Java does not have a good mechanism for modifying environment variables. EnvWriter is an attempt to workaround this, by using Reflection API, and modifying the UnmodifiableMap which holds the variables. This technique is now discouraged by Oracle. Good reads on this topic: blog post by Oracle

EnvWriter is mainly needed to support _X_AMZN_TRACE_ID which is a dynamic variable and could potentially change for every request. The downsides of this implementation are

• RIC is not compilable with the new Java versions
• nasty warning messages when accessing internal module with Reflection API
• makes GraalVM support more complicated

I feel like the Context object is a better home for the dynamic properties of an event. Moving _X_AMZN_TRACE_ID to the Context, and removing EnvWriter completely would simplify the codebase, make it healthier, and will improve compatibility with the recent Java releases.

It would be good to hear from the community. How much effort would it require for you to migrate away from the environment variable to use a property in the Context? Other thoughts on the topic are welcome :-)

[faermanj]

@smirnoal I agree that moving _X_AMZN_TRACE_ID but not so much removing EnvWriter completely. Most frameworks, inclusing spring boot and quarkus, can be configured from environment variables that are automatically mapped to configuration properties.

[smirnoal]

Good point @faermanj The frameworks would still have access to all environment variables which Lambda defines.

Except for _X_AMZN_TRACE_ID, which would be available from Context