Support batches with > 10 messages

[VictorJuliani]

Is your feature request related to a problem? Please describe.

Yes. For a while now, it's possible to receive batches with more than 10 entries in Lambda (ref). However, the calls from SendMessageBatch and DeleteMessageBatch in AWS PowerTools are made with the entire set of messages that are being sent to the DLQ or deleted from the SQS.

Those APIs, however, are limited to 10 entries each, which results in exceptions like TooManyEntriesInBatchRequestException: Maximum number of entries per request are 10. You have sent 226.

Describe the solution you'd like

Batch API calls should be made in chunks of 10 to satisfy the SDK restrictions.

Describe alternatives you've considered

So far, reducing the batch size to 10, which is not ideal for my solution due to the amount of concurrent lambdas.

[pankajagrawal16]

Thanks for opening the issue. I believe we made a similar fix in python powertools as well. I will look into fixing it this week.

[heitorlessa]

We’ve just added support yesterday for the new native partial response that is a much better alternative now https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/batch/ I’ve documented the mechanics, events, responses and I posted the details on the RFC too (on roadmap repo)

…

On Tue, 21 Dec 2021 at 16:07, Pankaj Agrawal ***@***.***> wrote: Thanks for opening the issue. I believe we made a similar fix in python powertools as well. I will look into fixing it this week. — Reply to this email directly, view it on GitHub <#663 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBBVKTSPRUFRF3VOYM3USCJZLANCNFSM5KN6IYUQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[pankajagrawal16]

Yeah. This one is for Java which does not have the new batch implementation yet, But this is a bug/limited functionality in older util

[heitorlessa]

Yeah, for the older one we still have to fix that too - we have a PR at an okay stage but we’re fixing it for the next release. We’ll add a warning for them to move To the new one as their cost and latency will decrease too

…

On Tue, 21 Dec 2021 at 16:16, Pankaj Agrawal ***@***.***> wrote: Yeah. This one is for Java which does not have the new batch implementation yet :p .. but this is a bug/limited functionality in older util — Reply to this email directly, view it on GitHub <#663 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBFBFN6R5WLKV3YPHKLUSCK4RANCNFSM5KN6IYUQ> . You are receiving this because you commented.Message ID: ***@***.***>

[pankajagrawal16]

Yeah, for the older one we still have to fix that too - we have a PR at an okay stage but we’re fixing it for the next release. We’ll add a warning for them to move To the new one as their cost and latency will decrease too
…
On Tue, 21 Dec 2021 at 16:16, Pankaj Agrawal @.> wrote: Yeah. This one is for Java which does not have the new batch implementation yet :p .. but this is a bug/limited functionality in older util — Reply to this email directly, view it on GitHub <#663 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZPQBFBFN6R5WLKV3YPHKLUSCK4RANCNFSM5KN6IYUQ . You are receiving this because you commented.Message ID: @.>

Ah ok.. so thats not merged yet in python as well. Good to know. I will look if I can get this fix out first and then plan for newer version post that.

[pankajagrawal16]

Related python issue aws-powertools/powertools-lambda-python#824

[pankajagrawal16]

Available in https://github.com/awslabs/aws-lambda-powertools-java/releases/tag/v1.10.0