Xray instrumentor for SDK V2 no longer works due to conflict in "execution.interceptors" file

[MamishIo]

Describe the bug

From https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java.html: it should be possible to add the com.amazonaws:aws-xray-recorder-sdk-aws-sdk-v2-instrumentor dependency to automatically instrument SDK V2 clients with Xray trace information. This no longer works in newer Java SDK V2 versions due to recently added conflicting resources.

Expected Behavior

Adding dependency "aws-xray-recorder-sdk-aws-sdk-v2-instrumentor" should automatically instrument SDK clients to produce useful Xray traces.

Example screenshot from Xray service map showing tracing-enabled Lambda function calling STS, on SDK version 2.15.0:

Current Behavior

SDK clients as of 2.15.35 (and possibly earlier) are no longer instrumented automatically. Same Lambda function above no longer instruments STS call:

Steps to Reproduce

Example repo: https://github.com/MamishIo/aws-sdk-java-v2-instrumentor-repro
Setup: mvn package to build "target/aws-sdk-java-v2-instrumentor-repro-1.0-SNAPSHOT-jar-with-dependencies.jar"

Quick repro:

1. Open JAR as ZIP and verify that file software/amazon/awssdk/global/handlers/execution.interceptors contains only one line software.amazon.awssdk.core.internal.interceptor.HttpChecksumRequiredInterceptor.
2. Change SDK version in POM to 2.15.0 and run mvn package again. Check same file in ZIP and verify that it now contains only one line com.amazonaws.xray.interceptors.TracingInterceptor.

Full/detailed repro:

1. Create Lambda function (handler=io.mamish.example.LambdaHandler, tracing=active, above JAR as code package).
2. Run with dummy Lambda test event (must be JSON string e.g. "test"). Use Xray console to verify function invocation shows but STS API call does not.
3. Change SDK version in POM to 2.15.0 and run mvn package again. Upload JAR to function.
4. Run Lambda function again and verify that Xray console shows function invocation and STS API calls linked together in service map.

Possible Solution

Appears to have been caused by commit 9cb352e which added file software/amazon/awssdk/global/handlers/execution.interceptors (ref) to define a new standard interceptor. This path was already being used by the instrumentor (ref) to set a global Xray interceptor but this is now being overridden by the SDK's file.

Not sure what the ideal solution is, but should be possible to work around temporarily by adding a custom execution.interceptors file containing both class names.

Improving ClasspathInterceptorChainFactory to read from multiple interceptors files could be a better long-term solution.

Context

Impact: My architecture has services invoking each other using the SDK (i.e. mainly Lambda). Automatic instrumentation made it easy to trace a request between services, but this no longer works on new SDK versions since trace information isn't propagated.

Your Environment

• AWS Java SDK version used: Failing on 2.13.15, working on 2.13.0
• JDK version used: default for Corretto 11 Lambda runtime
• Operating System and version: default for Corretto 11 Lambda runtime

[debora-ito]

@MamishIo thank you for the detailed report. We will investigate.

[millems]

The SDK supports multiple execution.interceptors files in different JARs, but it looks like you're creating a bundled JAR, so the bundling process must be dropping one.

Can you try using the Maven shade plugin to create the bundled jar? That will allow you to use the AppendingTransformer to merge these files, instead of override one with the other. https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#AppendingTransformer

I'm not sure if the maven assembly plugin you're using supports a similar feature.

<project>
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.2.4</version>
        <executions>
          <execution>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                  <resource>software/amazon/awssdk/global/handlers/execution.interceptors</resource>
                </transformer>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  ...
</project>

[millems]

Created a PR to illustrate what the migration might look like in your repro application: MamishIo/aws-sdk-java-v2-instrumentor-repro#1

[MamishIo]

Thanks @millems , I appreciate the demo PR a lot. Assembly doesn't appear to have any equivalent to Shade's transformers so I'll just have to switch plugins.

Will go ahead and close the issue since it's definitely a side effect of the build process rather than anything wrong with the SDK. Thanks again!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.