Skip to content

fix: retry on stale ETag #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 26, 2020
Merged

Conversation

hoonoh
Copy link
Contributor

@hoonoh hoonoh commented Oct 12, 2020

I kept facing this error from time to time:

arn:aws:lambda:us-east-1:*****:function:*****:91 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:91 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:21 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:26 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:18 is associating to ***** CloudFront Distribution. waiting for deployed status.
Serverless: Recoverable error occurred (Rate exceeded), sleeping for ~5 seconds. Try 1 of 4
arn:aws:lambda:us-east-1:*****:function:*****:58 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:13 is associating to ***** CloudFront Distribution. waiting for deployed status.
Serverless: Recoverable error occurred (Rate exceeded), sleeping for ~5 seconds. Try 1 of 4
 
  Serverless Error ---------------------------------------
 
  The request failed because it didn't meet the preconditions in one or more request-header fields.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              12.18.4
     Framework Version:         1.82.0 (local)
     Plugin Version:            3.8.3
     SDK Version:               2.3.1
     Components Version:        2.34.9

Did some googling and found:
https://www.pulumi.com/docs/tutorials/aws/aws-ts-static-website/#preconditionfailed-the-request-failed-because-it-didn-t-meet-the-preconditions

This is caused by CloudFront confirming the ETag of the resource before applying any updates. ETag is essentially a “version”, and AWS is rejecting any requests that are trying to update any version but the “latest”.

So I am guessing this error happens when multiple function updates are requested into single Cloudfront distribution consecutively.

This patch wraps config generation routine (which includes (re)fetching latest CF ETag) and updateDistribution request into an async function and retries if faced with PreconditionFailed error (max 5 retries with 5 seconds delay).

With this patch, the above error is handled as followed:

arn:aws:lambda:us-east-1:*****:function:*****:97 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:97 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:22 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:27 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:19 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:64 is associating to ***** CloudFront Distribution. waiting for deployed status.
arn:aws:lambda:us-east-1:*****:function:*****:14 is associating to ***** CloudFront Distribution. waiting for deployed status.
received precondition failed error, retrying... (5/5)
Done in 141.27s.

Copy link
Contributor

@horike37 horike37 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense 👍
Thanks!

@horike37 horike37 merged commit 19dc0b6 into serverless-operations:master Oct 26, 2020
@horike37
Copy link
Contributor

🎉 This PR is included in version 1.1.5 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants