Maintenance: Refactor artifact name strategy for parallel uploads

[heitorlessa]

Why is this needed?

GitHub actions/upload-artifact v4 has a racing condition when uploading artifacts with the same artifact name in parallel.

For example, our Lambda Layer CDK artifacts are uploaded on a per region deploy in parallel using the a static output name.

This caused today's release to fail partially - re-running failed jobs served as a workaround.

Which area does this relate to?

No response

Solution

Since it's a race condition on upload, we need a dynamic artifact name e.g., one per region. Then merge these artifacts on download.

Updating to a dynamic name

      - name: Save Layer ARN artifact
        if: ${{ inputs.stage == 'PROD' }}
        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4.3.1
        with:
          name: cdk-layer-stack-{{ matrix.region }}
          path: ./layer/cdk-layer-stack/* # NOTE: upload-artifact does not inherit working-directory setting.
          if-no-files-found: error
          retention-days: 1
          overwrite: true   # still needed to allow failed retries

Downloading all artifacts for all regions

      - name: Download CDK layer artifact
        uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe # v4.1.2
        with:
          name: cdk-layer-stack
          path: cdk-layer-stack/
          merge-multiple: true

We'll need a testing environment to create a parallel job to test this out.

Acknowledgment

• This request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[heitorlessa]

linking a sub-action solution if we also need it in the future: actions/upload-artifact#505

[heitorlessa]

Working on this tomorrow morning so we can make a patch release

[dreamorosi]

We're also planning on making a release in the afternoon; let us know if you want to team up or just bounce ideas.

[heitorlessa]

Let’s team up, it’s easier ;)

…

On Mon, 19 Feb 2024 at 20:32, Andrea Amorosi ***@***.***> wrote: We're also planning on making a release in the afternoon; let us know if you want to team up or just bounce ideas. — Reply to this email directly, view it on GitHub <#3748 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBE26GDFPTCNEZXLDVDYUOSF5AVCNFSM6AAAAABDBOU2IKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJTGA3DCNRYHA> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>