Timing issue when creating bucket right after user

[deen13]

The PutBucketPolicyRequest fails in our process at the first attempt with a Invalid principal in policy error. This seems like a timing issue because the referenced user is created right before the PutBucketPolicyRequest which succeeds after a couple of retries.

Exception in thread "vert.x-eventloop-thread-7" software.amazon.awssdk.services.s3.model.S3Exception: Invalid principal in policy (Service: S3, Status Code: 400, Request ID: 72DC1788979CCEFC, Extended Request ID: I2C3iXnFi259I235bZstN55k8dZBtQKwjLh/XStWEM17aNyS4Tui6A1UkTXenIxxASw2u+gpZxg=)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42)
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$4(BaseClientHandler.java:214)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:88)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:129)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:179)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$500(ResponseHandler.java:69)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:295)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.publishMessage(HandlerPublisher.java:402)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.flushBuffer(HandlerPublisher.java:338)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.receivedDemand(HandlerPublisher.java:291)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.access$200(HandlerPublisher.java:61)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:834)

Steps to Reproduce

1. Create IAM User

val createUserRequest = CreateUserRequest.builder().userName("my-user").build()
val user = iamClient.createUser(createUserRequest).await().user()

2. Create S3 Bucket with Private Canned ACL

val createBucketRequest = CreateBucketRequest.builder()
            .bucket("my-bucket")
            .acl(BucketCannedACL.PRIVATE).build()

s3Client.createBucket(createBucketRequest).await()

3. Grant Access to the Bucket using a Policy

val policy = """ 
                {
                   "Version":"2012-10-17",
                   "Statement":[
                      {
                         "Sid":"AllowObjectReadWrite",
                         "Effect":"Allow",
                         "Principal":{
                            "AWS":"${user.userId()}"
                         },
                         "Action":[
                            "s3:GetObject",
                            "s3:PutObject"
                         ],
                         "Resource":[
                            "arn:aws:s3:::my-bucket/*"
                         ]
                      },
                      {
                         "Sid":"AllowBucketList",
                         "Effect":"Allow",
                         "Principal":{
                            "AWS":"${user.userId()}"
                         },
                         "Action":[
                            "s3:ListBucket"
                         ],
                         "Resource":[
                            "arn:aws:s3:::my-bucket"
                         ]
                      }
                   ]
                }
            """.trimIndent()

val policyRequest = PutBucketPolicyRequest.builder()
    .bucket(bucketName(system))
    .policy(policy)
    .build()

return s3Client.putBucketPolicy(policyRequest).await()

Workaround

The third, step which contains the PutBucketPolicyRequest, succeeds if we use a retry mechanism.

retry(limitAttempts(5) + constantDelay(5_000)) { s3Client.putBucketPolicy(policyRequest).await() }

Context

We're working on a cloud platform that manages parking systems. A business requirement is that each parking system can stream its local database backups to a s3 bucket. In order to accomplish that goal a user and a bucket is created for each parking system during the provisioning process.

Your Environment

• AWS Java SDK version used: 2.13.51
• JDK version used: 11.0.2
• Operating System and version: macOS Mojave 10.14.6

[debora-ito]

Hi @deen13 every resource change can take some time to propagate, it takes time for the information to be reflected globally in the AWS infrastructure. The solution for this issue is Waiters but they are not available yet in Java v2, we are tracking the feature request in #24 (the linked blog post is for Java v1).

Let us know if that helps.

[deen13]

I switched back to the V1 to give it a shot but I don't know for which resource we exactly need to wait. The S3-Waiter returns after 30 milliseconds when using it with a HeadBucketRequest. 😕

val user = awaitBlocking { iamClient.createUser(CreateUserRequest("my-user")).user }

awaitBlocking {
    iamClient.waiters().userExists().run(WaiterParameters(GetUserRequest().withUserName(user.userName)))
}

awaitBlocking {
    s3Client.createBucket(CreateBucketRequest("my-bucket").withCannedAcl(Private))
}

awaitBlocking {
    s3Client.waiters().bucketExists().run(WaiterParameters(HeadBucketRequest("my-bucket")))
}

val ls = Statement(Statement.Effect.Allow)
    .withPrincipals(Principal(user.userId))
    .withActions(S3Actions.ListObjects)
    .withResources(Resource("arn:aws:s3:::my-bucket"))

val rw = Statement(Statement.Effect.Allow)
    .withPrincipals(Principal(user.userId))
    .withActions(S3Actions.GetObject, S3Actions.PutObject)
    .withResources(Resource("arn:aws:s3:::my-bucket/*"))

awaitBlocking { s3Client.setBucketPolicy("my-bucket", Policy().withStatements(rw, ls).toJson()) }

[debora-ito]

Some services like S3, EC2 and DynamoDB have documented their consistency models. The IAM user guide also mentions they use Eventual Consistency model and offers some guidelines:

You must design your global applications to account for these potential delays. Ensure that they work as expected, even when a change made in one location is not instantly visible at another. Such changes include creating or updating users, groups, roles, or policies. We recommend that you do not include such IAM changes in the critical, high-availability code paths of your application. Instead, make IAM changes in a separate initialization or setup routine that you run less frequently. Also, be sure to verify that the changes have been propagated before production workflows depend on them.

We are working on the Waiters for SDK v2, you can check the design doc (#1875) and follow the updates in the tracking issue.

[deen13]

@debora-ito thanks for the background information but what went wrong with my latest code snippet? 😕 I thought the workers would help me to deal with the eventual consistency.

[debora-ito]

I'm sorry, it was not clear for me that something was wrong. What's the issue exactly?