GetObjectRequest with requestPayer set to RequestPayer.REQUESTER generates invalid presigned URL

[philvarner]

Describe the bug

Signing a GetObjectRequest that has requestPayer set to RequestPayer.REQUESTER produces a URL that, when executed using GET, fails.

Expected Behavior

When I sign a GetObjectRequest that has requestPayer set to RequestPayer.REQUESTER, I get a URL that successfully retrieves the object with a GET command.

Current Behavior

When I sign a GetObjectRequest that has requestPayer set to RequestPayer.REQUESTER, I get a URL that results in an error "SignatureDoesNotMatch" The request signature we calculated does not match the signature you provided. Check your key and signing method. An example of a URL like this is:

https://astraea-opendata.s3.us-west-2.amazonaws.com/MCD43A4.006/25/06/2001055/MCD43A4.A2001055.h25v06.006.2016113010159_B01.TIF?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200824T170032Z&X-Amz-SignedHeaders=host%3Bx-amz-request-payer&X-Amz-Expires=300&X-Amz-Credential=AKIAITO3BRCS3G5FTUNA%2F20200824%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=37d0602d15e66baba8d18a79f97f5b1e60ef504299c5f68e623bf07a1ea8babc

Steps to Reproduce

The code (Scala) to used generate this is below, not the call to requestPayer

    val getObjectRequest = GetObjectRequest
      .builder()
      .bucket("astraea-opendata")
      .key("MCD43A4.006/25/06/2001055/MCD43A4.A2001055.h25v06.006.2016113010159_B01.TIF")
      .requestPayer(RequestPayer.REQUESTER)
      .build()
    val getObjectPresignRequest = GetObjectPresignRequest
      .builder()
      .signatureDuration(java.time.Duration.ofMinutes(5))
      .getObjectRequest(getObjectRequest)
      .build()
    val presigner = S3Presigner.builder().region(Region.US_WEST_2).build()
    val url = presigner.presignGetObject(getObjectPresignRequest).url()
    println(url.toExternalForm)

However, if one removes the requestPayer call and replaces it with:

      .overrideConfiguration(
        AwsRequestOverrideConfiguration
          .builder()
          .putRawQueryParameter("x-amz-request-payer", "requester")
          .build()
      )

The a url that works for retrieving the object is returned, e.g.,

https://astraea-opendata.s3.us-west-2.amazonaws.com/MCD43A4.006/25/06/2001055/MCD43A4.A2001055.h25v06.006.2016113010159_B01.TIF?X-Amz-Request-Payer=requester&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200824T172144Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=AKIAITO3BRCS3G5FTUNA%2F20200824%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=ad686e9dcc59469f91389fce00fa8f87b7b4583a2fb30b36905bead5a46af133

Possible Solution

I believe the underlying problem here is that GetObjectRequest.requestPayer sets a header for X-Amz-Request-Payer by default, so the resulting signed URL must be called with this header for it to work. However, there's no documentation for this anywhere and for any use of signed URLs by web browsers, the browser can't be forced to add that header. The documentation for presignGetObject indicates that it's a URL you can just GET to retrieve the object, with no mention of additional headers, etc. you may need to set.

I believe the correct behavior here is when requestPayer is set on a signed URL to put this as a query parameter in the request and sign it as such.

Context

The use case I have is to provide https urls to S3 objects that are in a public requester-pays bucket I own, to users of web service I have.

Your Environment

• AWS Java SDK version used:2.14.2
• JDK version used: 13
• Operating System and version: MacOS 10.16.6 Catalina

[debora-ito]

Hey @philvarner I think you answered your own issue. S3 docs says:

For signed URLs, include x-amz-request-payer=requester in the request

So setting as a query param seems to be the way to do it. I know we have a different issue in presigned URL opened for the x-amz-acl header, and it also worked as a query param: #1849

The S3 documentation also mentions that "Bucket owners do not need to add x-amz-request-payer to their requests." I don't know if you are signing the URLs with credentials of the bucket owner in your use case, but it's worth a try.

Let us know if that helps.

[philvarner]

Hi @debora-ito, thanks for the reply. You're right, the S3 REST API docs do say that, but there's nothing in the Java SDK v2 docs that indicate how you would create a GetObjectRequest the has that set. I think this is a variation on #1849, and the underlying problem is that there is no documentation that requestPayer and cannedAcl won't be reflected in the signed url, and are no examples of how to use putRawQueryParameter at all, much less specific to these parameters. Having examples would really help.

Also, I don't think i was clear enough in the description that setting the parameter using putRawQueryParameter does work correctly, even on public requester-pays buckets for which I'm not the owner.

Also, in the initial report (now corrected), I stated that the query parameter name should be "A-Amz-Request-Payer", but it actually has to be lower-kebab-case "x-amz-request-payer" or it doesn't work for non-owner buckets. This is confusing because all of the other query string parameters are Upper-Kebab-Case e.g. "X-Amz-Algorithm", but this and x-amz-acl and I assume others have to be lower.

Hope that describes the desired documentation well enough!

[debora-ito]

Yes that's fair, we can definitely improve our documentation. Changing this to a documentation issue.