Current state: pausing new features until end of July to focus on operational excellence

[heitorlessa]

Background

AWS Lambda Powertools is a suite of utilities to ease adopting best practices defined in the Well-Architected Serverless Lens. Our mission is to help customers implement these practices as early in their software development cycle as possible. We carefully crafted our Tenets to help govern our decision making, as we make progress towards our mission.

The AWS staff maintaining this project do so in their spare time. A small group of AWS staff volunteers, including Solutions Architects and Professional Services Consultants, governs each Lambda Powertools language separately. We meet regularly to share lessons learned across languages to improve the experience for everyone. This helps us stay focussed on the end-to-end user experience for each language while considering that customers may use multiple languages and AWS services.

Growth

Lambda Powertools for Python became general available in June 2020 with three core features: tracing, structured logging, and asynchronous metrics. Since then, we have launched Lambda Powertools for Java, 11 new utilities for Lambda Powertools for Python, and more recently, a beta for Lambda Powertools for TypeScript.

For Python, we have experienced unprecedented growth, going from 5K to ~800K downloads per month in less than 20 months. This rapid growth has led to a vibrant community building effort. Community submissions now contribute, on average, 50-60% to each release. These contributions vary from documentation, features, bug fixes, Request for Comments (RFC), and blog posts.

It takes a significant amount of our spare time to triage, discuss, review, answer follow-up questions, and merge contributions. As customer activities in our daily jobs increases, our spare time became even scarcer. As with any successful, fast-growing project, we have accumulated technical and project management debt.

As a result, we need to change how we work to make it sustainable for everyone involved: customers, community, and maintainers.

Moving forward

For the AWS Lambda Powertools for Python, we are shifting our focus to Operational Excellence, prioritizing stabilization over new features and enhancements. This focus period will last until the end of July 2022. During this time, we are pausing external contributions as we work on more sustainable mechanisms to communicate and collaborate with the community. We will continue to support bug reports but no longer feature requests until the end of this period.

This allows us to focus on the following activities:

• Operational Excellence. Lambda Powertools for Python has an estimated 6.5 million downloads (as of Feb 2022) with 200K weekly downloads, and hundreds of customers using it in production. This means reliability is at the core of what we do, regardless of this pause.
• Roadmap. Experiment with the new GitHub Project Beta. This allows us to offer an improved view of our current priorities, backlog, ideas, and more easily emphasize areas we need for customer feedback. Centralize cross-languages concerns to expose expected vs optional feature parity across them. Revise the use of GitHub Discussions over Slack to improve discoverability and long-running discussions
• Refresh contribution guidelines. Ease new contributor on-boarding by automating cloud developer environments (i.e. GitPod.io). Provide detailed guidance on mechanisms to triage and prioritize all community contributions. Clarify requirements for bug reports. Describe circumstances when a maintainer may close or reopen an issue/pull request.

We wish to acknowledge how extremely grateful we are for the work of several individuals who have helped shape Lambda Powertools into what it is today. Your contributions within and outside GitHub help thousands of customers adopt best practices when building serverless applications.

We appreciate your understanding as we focus on this stabilization period. We can't wait to get back to accepting external feature contributions again in a way that responsibly looks after our maintainers' mental well-being.

Kindly, the maintainers of AWS Lambda Powertools for Python.

[heitorlessa]

We would like to share some of the progress we've made around Roadmap and Contributions outlined above.

• Roadmap. We began our central roadmap migration to a new dedicated board of activities. This board aims to more accurately represents our main focus, and to more easily identify issues that are blocked due to pending customer interest, pending RFCs, what's coming soon in the next release, etc.
• Future repurpose of central roadmap. We started working on an internal discussion about standardizing a baseline of labels and processes that will be common across each Lambda Powertools language (Java, TypeScript). As we learn the quirks and automation mechanisms of the new GitHub Projects Beta, we will be migrating all roadmap items to their respective repository. Once this completes, we will repurpose the central roadmap repository to become a landing page for all Lambda Powertools languages, where you will find feature parity across them.
• Refresh contribution guidelines. We've published our first iteration in documenting the roadmap process. We've started working on a MAINTAINERS.md document. It'll describe our responsibilities, iterate on our processes, make it easier for customers to identify who are the official maintainers (@heitorlessa, @am29d, @mploski, @sthulb), etc. As we settle on our first iterations for both roadmap and maintainers process, we will be updating the CONTRIBUTING.md document guide as promised. The goal is to ensure we a) focus on making it easier to contribute (dev setup), b) set expectations for each type of contribution (docs, bugs, features, etc.), and c) provide a new mechanism to more easily have maintainers to mentor new and existing contributors to fast track their contributions.

[heitorlessa]

New updates:

• Refresh contribution guidelines. We've published our first iteration of the Maintainers playbook. This formalizes the process maintainers follow to triage issues, PRs, governance in general, who they are, and the label strategy used throughout this project.
• Operational Excellence. @mploski completed the first draft of E2E testing RFC and started its implementation. After completing governance automation tasks, we will start working on integration testing in parallel to @mploski E2E work. @am29d made progress on Lambda Layer release pipeline migration and started experiment with expanding to all 23 commercial regions, including new public Lambda Layers for ARM and with extra dependencies (Parser).

[heitorlessa]

Today marks the end of our pause for new features 🎉. This means that triaged feature requests and RFCs are now considered - we welcome contributions from the community granted that authors of feature requests/RFCs have the 1st priority.

We deeply appreciate everyone's patience while we worked on a number of areas and releases (below), and helped our colleagues with the exciting launch of Lambda Powertools for TypeScript GA. While we will continue to work on many of these internal areas, we're equally excited to get back on improving everyone's developer experience.

Last but not least, please help me welcome our new full-time maintainer @rubenfonseca 🥰 - one of many great news we can't wait to share with you this month alone.

[1] Releases since the pause started:

• 1.25.6
• 1.25.7
• 1.25.8
• 1.25.9
• 1.25.10
• 1.26.0
• 1.26.1
• 1.26.2
• 1.26.3
• 1.26.4
• 1.26.5
• 1.26.6
• 1.26.7

Details

Operational Excellence

End-to-end testing

We've completed the first steps of our internal E2E test framework and covered Tracer, Logger, and Metrics. As we began to improve E2E coverage, we realized we should do a lot more to make it easier to write these tests - both for maintainers and avid contributors. Once that settled, we'll publish our definition of functional vs integ vs E2E tests in the new contribution guidelines.

We will continue working on this regardless of the end our pause for new features.

Release improvements

Our release process is now down to one manual step: Draft release notes. We found it to be a good balance between making releases as effortless as possible while keeping the human touch. Our goal remains to ensure everyone understands the value and how to best utilize each release, including highlighting every contribution received no matter the size.

Lambda Layers

We've completed the migration to GitHub Actions and expanded to all AWS Commercial regions. This reduced our immediate availability of new Lambda Layers to all regions within 5 minutes after PyPi release.

We will continue working on this task. We are looking for ways to automate documentation generation for all 22 Lambda Layer ARNs. Once complete, we can create additional layers for example Extras, and ARM support.

Revamp GitHub Actions automation

We refactored all GitHub Actions using inline scripts (simple tasks or not) into standalone GitHub-Script, and started moving into composable workflows with the new reusable workflows feature - this reduced 1/3 of our operational overhead.

Additionally, we audited our use of GitHub Actions and followed GitHub Security Guide practices where applicable.

Roadmap

Roadmap migration

We migrated all roadmap and day-to-day activities to the new GitHub Projects - here. The Themes/Epics view should give everyone a glimpse of groups of activities are our main focus, and how further along they are.

We also publicly documented the roadmap process. It gives an overview on key groups of activities, our status definition, and the process each maintainer will follow.

Refresh contribution guidelines

Maintainers process

We published our first maintainer's playbook. The intention is to provide more details about how each maintainer is expected to interact with the community, and process they'd keep in mind when doing so.

We also now enforce that PRs have a related issue to avoid significant delays, and to account for a balance on customer demand, scope, and maintenance effort. This also automates the process of notifying customers when their change was released with a given version.

Pre-configured Cloud based contributor's environment

We had limited success in providing a pre-configured GitPod.io instance for contribution due to issues with the pre-commit framework and GitPod's security model.

We will continue working on this and break down all tasks in our board to lower the entry bar to contribution.

New issue templates

We created 3 new issue templates: Static typing (Mypy related), Ask a question (GitHub Discussions), and Maintenance (tech debt related). This also means we removed the option for blank issues to ensure any non-issue remains in GitHub Discussions for helpful threaded conversations.