Bug: parameters.get_parameters(.., max_age=24*60*60, force_fetch=True) does not work as expected anymore

[NastHoloviz]

Expected Behaviour

There was the issue found that when you grab data with force_fetch=True from the code you still get old value from the cache instead.
You can find the example of the code of AWS Lambda lambda function in the code snippet as well, but I will explain here the details.
So here are some steps:

1. Get parameters with parameters.get_parameters(.., max_age=24*60*60, force_fetch=False) - this will either get parameters data from API and put them into the cache or get parameters from cache if lambda was triggered before already (lambda function env memory).
2. Update one of the parameters within same lambda function code (you can make sure later going to AWS Console that parameter was really updated).
3. Get parameters with parameters.get_parameters(.., max_age=24*60*60, force_fetch=True) within same code of same lambda function and see that value for updated parameter is still the old one. So that means that force_fetch actually either did not get updated value from the API or did not update the cache.

In the context of one lambda function execution it might be useless, but if we get params in lambda and put them to the cache, in couple minutes update some of the parameters and then trigger lambda function again with force_fetch=True - this lambda function still keeps old value in the memory. This is the issue.
So expected behavior is that with force_fetch=True we will always get values updated from SSM.

Current Behaviour

1. Get parameters with parameters.get_parameters(.., max_age=24*60*60, force_fetch=False) - this will either get parameters data from API and put them into the cache or get parameters from cache if lambda was triggered before already (lambda function env memory).
2. Update one of the parameters within same lambda function code (you can make sure later going to AWS Console that parameter was really updated).
3. Get parameters with parameters.get_parameters(.., max_age=24*60*60, force_fetch=True) within same code of same lambda function and see that value for updated parameter is still the old one. So that means that force_fetch actually either did not get updated value from the API or did not update the cache.

Code snippet

@app.lambda_function()
def test_force_fetch(event: dict, context: object):
    """Lambda function to test force_fetch=True is not working."""
    # Get params from SSM, use cache here as well
    params = parameters.get_parameters(
        path="/path-to-param/",
        recursive=True,
        decrypt=True,
        max_age=24 * 60 * 60,  # 24 hours
        force_fetch=False,
    )
    config = {}
    for key, value in params.items():
        config[key] = json.loads(value)
    app.log.info("Test value: %s", config["TEST_VALUE"])

    # Update SSM parameter manually
    ssm = boto3.client("ssm")
    new_value = str(uuid.uuid4())
    app.log.info("Value to set in SSM: %s", new_value)
    ssm.put_parameter(
        Name="/path-to-param/TEST_VALUE",
        Value=json.dumps(new_value),
        Type="SecureString",
        Overwrite=True,
    )
    time.sleep(5)

    # Get params from SSM with force_fetch, use cache here as well
    params = parameters.get_parameters(
        path="/path-to-param/",
        recursive=True,
        decrypt=True,
        max_age=24 * 60 * 60,  # 24 hours
        force_fetch=True,
    )
    config = {}
    for key, value in params.items():
        config[key] = json.loads(value)
    app.log.info("Test value with force: %s", config["TEST_VALUE"])

Possible Solution

No response

Steps to Reproduce

Please see Code snippet and Expected Behaviour sections.

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.9

Packaging format used

PyPi

Debugging logs

Please see logs in attached screenshot.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hey @NastHoloviz, thanks for reporting this! I'm taking a look in this issue right now.

[leandrodamascena]

Hi @NastHoloviz! Again, thanks a lot for reporting this, yes I can confirm this is a bug when using get_parameters, but using get_parameter it works as expected. I'm working on submitting a fix for this.

[leandrodamascena]

Python shines brightly when it’s at its best, but it can be a real headache when it’s not. In the get_multiple function the order of parameter force_fetch and raise_on_transform_error is inverted, and since the function from the main class expects `**kwargs, it was failing. Sending a PR.