Feature request: support clearing/scoped caches for testing

[huonw]

Use case

We're testing our code with mocked AWS services using moto (rather than replacing/patching the calls entirely), and it works well. However, powertools provides various caches that are great for production, but cause unintended linkage/interference between tests. We're hitting this particularly in aws_lambda_powertools.utilities.parameters.get_secret, but it applies to other parts of that module, and likely any other global caches built into this package.

It'd be nice to be able to run tests isolated without having to manually write force_fetch=running_in_tests() or similar everywhere.

Solution/User Experience

Some method for clearing caches. For instance, a global aws_powertools_lambda.clear_caches() or aws_powertools_lambda.utilities.parameters.clear_caches(). Tests could then have set-up that calls that to ensure each one has a fresh environment.

Alternative solutions

Alternatives:

• an environment variable that disables caching entirely, which can be set during tests runs (not so good because the deployed behaviour differs)
• using contextvars to have the "global" caches actually be scope-able (this seems complicated)

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[huonw]

As workaround for now we've got a pytest fixture that pokes into the internals of this package:

import aws_lambda_powertools.utilities.parameters.base

# FIXME: https://github.com/awslabs/aws-lambda-powertools-python/issues/1092
@pytest.fixture(autouse=True)
def clear_powertools_cache() -> None:
    for v in aws_lambda_powertools.utilities.parameters.base.DEFAULT_PROVIDERS.values():
        v.store.clear()

[michaelbrewer]

Uh, @huonw that makes sense, i have run into the same kind of issue, and did implement force_fetch option.

[michaelbrewer]

This is closer to a bug, as testing should be something that is easier to do for parameters feature.

[heitorlessa]

thanks for flagging it @huonw :) @mploski is working on our end-to-end tests POC, and we should be able to address it as part of a much wider testing improvements we're looking to make.

Thanks a lot again

[heitorlessa]

Michal encountered a few performance challenges in the E2E POC which will delay working on this task. I'm gonna expedite it for today's release; PR incoming soon