lambda layer support for parser (and other optional features)

[mwarkentin]

Opening an issue as requested. We currently install lambda-powertools using the SAR layer method, however it appears that the optional dependencies to use advanced features like the parser are not available in the layer. We'd like a way to include these dependencies without having to maintain our own lambda layer pipeline (which would remove most of the value of installing the main powertools dependency using the SAR application).

Some possible approaches:

• Retain a single SAR application for the powertools dependency, but include all optional dependencies
  • Concerns: overall layer size? Not sure how big these dependencies are
• Configure a pipeline to publish multiple SAR apps with various combinations of optional dependencies
  • Concerns: Operational overhead for the AWS team, potential explosion of layer combinations (especially with 5-layer limit on lambda - unless there's some news about this at reinvent 😄 )
• Offload the work to all consumers of the powertools library
  • Concerns: Lots of repeated work, managing their own pipelines to generate layers, across all users of powertools

@heitorlessa are the extras required for this functionality included in the SAR app / lambda layer? Or do those need to be handled separately? Might be worth adding a note in the docs, not sure if that would be under the parser docs or the lambda layer installation docs?

Hey @mwarkentin - the latter, handled separately as described in the parser docs.

Unsure whether it worth creating a separate Layer for that tbh given the additional operational overhead. In V2, we'd like to create each utility to be pip installable if a theory works, then we could consider multiple layers I suppose.

Happy to discuss more in a separate issue tho ;)

Originally posted by @heitorlessa in #118 (comment)

[heitorlessa]

Hey @mwarkentin and @koiker - What do you think of having an additional layer with all optional dependencies installed?

Adding pydantic to the current Layer, parser's optional dependency, might break some existing customers as it increases the final Lambda package size by 70M.

Adding an additional layer with clear description that it'll contain all optional dependencies we might have (incl. in the future) allows customers who want to have an all-in experience, and it gives us room to think about 2.0 - potentially microlibs - where we can break them into categories as well as make them stand-alone.

[mwarkentin]

@heitorlessa that would work for us - do you think this would be a second layer included in the existing SAR application or a separate SAR application? 70MB is a decent size chunk I guess.. maybe those limits will be increasing soon? ;)

[koiker]

Sounds a good idea to have 2 packages: aws-powertools-core and aws-powertools-extras or aws-powertools-advanced. (I don't know, just trowing some ideas on the naming convention)

[heitorlessa]

Yep - That would be a second SAR App with an explicit name to differentiate between the one we have (similar to Rafael suggestion). In your case, you’d need only that new Layer instead of having to add multiple layers - this obviously would come at the expense of a larger package. If anyone in the future would like to only include certain optional dependencies they could always build their own Layer, or simply use their dependency package management tool

…

On Fri, 20 Nov 2020 at 22:05, Rafael M. Koike ***@***.***> wrote: Sounds a good idea to have 2 packages: aws-powertools-core and aws-powertools-extras or aws-powertools-advanced. (I don't know, just trowing some ideas on the naming convention) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#210 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBBX7XQD3ADOFOOQGWLSQ3K3NANCNFSM4T3PF5OA> .

[mwarkentin]

Sounds good!

[heitorlessa]

just agreed with @am29d - We're gonna work on this next @mwarkentin ;)

[mwarkentin]

Woot, excellent!

[heitorlessa]

we're hoping to get it released by EOD @mwarkentin - @am29d is on it ;)

[prasenjithaty]

@heitorlessa has this been released? I didn't see anything in the documentation.