Feature request: add AppConfig Extension support as AppConfig backend in Parameters utility

[whardier]

Key information

• RFC PR: (leave this empty)
• Related issue(s), if known: None
• Area: Parameters
• Meet tenets: Yes
• Approved by: ''
• Reviewed by: ''

Summary

Utilize the AWS provided AppConfig Lambda Layer. This layer is used to provide a proxy request to AppConfig configurations. The layer has built-in caching and automatically refreshes based on a TTL. This is important when utilizing the "full power" of AppConfig to more accurately provide configuration deployments to live resources as well as asses how many client-ids are covered by the deployment stages.

Layer: https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-integration-lambda-extensions.html

Motivation

Preloading configurations (during cold start) automatically and utilizing AppConfig as intended - ongoing deployment of configurables. Having a background task handle resource management while persisting a client-id is very helpful and having the background task that acts as a keepalive for that client-id regardless of request requirements will assist with deployment fact gathering.

Proposal

Utilize urllib.request to read from a formated URL composed of several environment variables related to the AppConfig lambda layer. Should be a fairly straight forward update to existing appconfig code in parameters module. Some thought needs to go in to manual refresh and opting out for specific requests.

If AppConfig parameters helpers exist in the other languages powertools then it should be a fairly straight forward change.

User Experience

Enable the AWS AppConfig Lambda Layer

The only corner case for configuration would be providing sane defaults to opt-in/opt-out of using the layers HTTP endpoint rather than boto3 directly.

Drawbacks

This could muddy some thoughts on how TTLs are handled between powertools and the layer software.

No additional deps.

Rationale and alternatives

Low impact of not including this. If appconfig parameters are pulled per request the keepalive provided by the lambda layer would be redundant.

Unresolved questions

...

[heitorlessa]

Hi @whardier thanks for creating a RFC, much appreciated!

We do this today in the Parameters utility - by default we cache any parameter(s) for 5s: https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/parameters/#adjusting-cache-ttl

This ensures we cache a given parameter(s) in memory and only make a call to AppConfig in the next invocation if the TTL has expired. There are also cases where you might want to always fetch the latest available value for a given parameter whether it's in cache or not: https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/parameters/#always-fetching-the-latest

Until today, we were not exposing the TTL adjustment to higher level functions which might have contributed to believing this wasn't possible.

In the next release, you'll be able to go from this:

from aws_lambda_powertools.utilities import parameters
from botocore.config import Config

config = Config(region_name="us-west-1")
appconf_provider = parameters.AppConfigProvider(environment="my_env", application="my_app", config=config)

def handler(event, context):
    # Retrieve a single secret
    value: bytes = appconf_provider.get("my_conf", max_age=500)

To this:

from aws_lambda_powertools.utilities import parameters

def handler(event, context):
    # Retrieve a single configuration, latest version
    value: bytes = parameters.get_app_config(name="my_configuration", max_age=500, environment="my_env", application="my_app")

Does this help?

Look forward to hearing from you

[whardier]

Actually I do utilize the underlaying caching (and am leveraging it in a tool @ work). The RFC is meant to support the direct use of the AWS AppConfig Lambda Layer in order to pre-load and let it handle the client related aspects of that interface.

From AWS docs:

If you use AWS AppConfig to manage configurations for a Lambda function, then we recommend that you add the AWS AppConfig Lambda extension. This extension includes best practices that simplify using AWS AppConfig while reducing costs. Reduced costs result from fewer API calls to the AWS AppConfig service and, separately, reduced costs from shorter Lambda function processing times.

Powertools addresses the caching aspect for most use cases. There are some differences between the layer and powertools in this respect.

Using powertools along with the layer, as an option, would promote more frequent instantiations of the configuration as pattern where fetching the most recent configuration data is just a dip to a localhost daemon that is managing the configuration data.

I can appreciate that the daemon also communicates with appconfig as a daemon external to the runtime and that daemon has several configurable factors addressing retries and cache time as well. The daemon persists the client id interaction with AppConfig and deployments have a tighter view of the rollout.

Overall this is a simple thing to handle using the base parameters class as an add on.

[heitorlessa]

Understood - that's helpful, thank you!

My initial concern was supporting Extensions as part of the project as it's not trivial to debug and maintain yet.

I'll come back to this the week after next, so I concentrate on the next release with an important feature.

[heitorlessa]

Added labels to express we'd like to hear customer feedback before considering it. We had the same discussion on other Powertools and we didn't get much traction from other customers wanting AppConfig Extension support -- we're experiencing the other way around (start with Extension then move to Powertools/Custom code etc.)