RFC: Switch to logback, SLF4J and logstash as logging framework

[roamingthings]

Key information

• RFC PR:
• Related issue(s), if known:
• Area: Logger
• Meet tenets: Yes

Summary

Switch to logback GitHub repository, SLF4J and logstash as logging framework used by the Logging module.

Motivation

Currently the Lambda Powertools are using log4j2 as logging framework. Log4j2 is a very popular and widely used logging library in the Java ecosystem. It offers a lot of features that are most likely not required by most Lambda functions. One of these features (JNDI support) has gained a lot of popularity through CVE-2021-44228.

Another popular logging framework that derived from log4j1 is [logback]((https://logback.qos.ch). When using logbook it's also required to use SLF4J as an abstraction layer. For example logback is the default logging framework used by Spring Boot. It has a smaller jar-artefact footprint than log4j2.

logstash is another library that adds JSON formatting and structured logging to logback.

This would allow applications to add structured logging in a clean and convenient way without adding and removing additional keys explicitly (see proposal for an example). This is important because it reduces boilerplate code and the business logic gets more visibility.

Proposal

Replace the current usage of log4j2 with logback, SLF4J and also add logstash.

The interface of the Logging module would not have to change much. However, the configuration of the layout has to be changed to use logback and logstash.

As stated in the motivation section the usage of logstash would allow for clean implementation when using structured logging.

For example

LoggingUtils.appendKey("foo", "bar");
log.info("Hello World!")
LoggingUtils.removeKey("foo");

would become

log.info("Hello World! {}", kv("foo", "bar"));

Beside this common use-case logstash offers other convenient methods to generate these structured arguments. Structured Logging with Structured Arguments for more details

Dependency analysis

License

Logback

Dual license model:
Eclipse Public License v1.0
GNU Lesser General Public License version 2.1

SLF4J

MIT license

Logstash

Apache 2

Releases

New releases on a regular basis

Community

PR activity on the project is moderate on both projects
Project maintainer take part in active community discussions for logback on GitHub
Jira for issue tracking and logstash on GitHub

Size of library and dependencies (current versions as of 2022-10-29)

Jackson dependency is already part of Lambda Powertools for Java and required for both approaches.

logback + SLF4J + Logstash

• slf4j-api-2.0.3.jar, 61.41 kB
• logback-classic-1.4.4.jar, 266.09 kB
• logback-core-1.4.4.jar, 576.91 kB
• logstash-logback-encoder-7.2.jar, 408 kB
  Sum: ~1.3 MB

Log4J2

• log4j-api-2.19.0.jar, 317.57 kB
• log4j-core-2.19.0.jar, 1.86 MB
  Sum: ~2.2 MB
  Please note that Lambda Powertools requires more Dependencies like Jackson for JSON layout serialization.

Drawbacks

Changing the logging framework would mean a breaking change for existing applications and also effect other modules of the Lambda Powertools for Java.

It may also be possible that organizations don't want/allow using logback or prefer log4j over logback for other reasons. This could be mitigated by SLF4J which offers a unified API to use either logback or log4j2. This would still be a breaking change for existing applications.

Rationale and alternatives

Unresolved questions

• Does logstash allow to define a layout that is similar to LambdaJsonLayout.json

[jeromevdl]

Thanks for your RFC @roamingthings and the deep analysis. It definitely makes sense to get rid of log4j. The question I have and that I'd like to investigate is: do we actually need to have another implementation or could we provide the same level of service with just slf4j and let the users choose their implementation. Because some developers are using log4j anyway so adding logback (and potentially logstash) would increase the package size for them. A library (like Lambda Powertools) should not bring a log implementation and force its users to inherit from this...

[roamingthings]

@jeromevdl I like your idea of only providing slf4j support and not forcing any framework to be used.
While writing this RFC this conclusion also came to my mind. So I would also prefer this option if it slf4j provides enough functionality required by the Logging Powertools. I could think of having documentation about how to integrate log4j, logback and/or logstash to support users.

[jeromevdl]

@pankajagrawal16 Hi Pankaj, I'm currently looking at this RFC and browsing our current logging module, I was not sure what PowertoolsResolverFactory and PowertoolsResolver are used for. Is it still useful, even with the JsonTemplateLayout?

[pankajagrawal16]

Hey @jeromevdl

Yeah its used for the layout config. Basically if you look at https://github.com/awslabs/aws-lambda-powertools-java/blob/master/powertools-logging/src/main/java/software/amazon/lambda/powertools/logging/internal/PowertoolsResolver.java

getname method in there with powertools is mapped as a resolver in json layouts under resources folder. That how log4j new way of configuring custom layout is. It also helped us making sure that when we moved to new way of layout, the log format and keys remained same as before to be backwards compatibility for customers and having to do minimal changes.

Let me know if it makes sense?

[pankajagrawal16]

Also, regarding this rfc, just my two cents. Supporting slf4j was heavily discussed during initial implementation and was decided to start with log4j2 as it practically support everything that slf4j has to offer and more.

That being said, it was also considered if there is community interest in future, we could easily plugin slf4j support without need to rewrite and get rid of log4j rather having another option for customer to pick one or the other.

This has benefits that the customers who are already heavily invested in log4j2 (which there are many) won't need to deal with any breaking change. Also if slf4j support is plugged inn, it reaches even wider set of customers without requiring them do alot of changes.

[driverpt]

IMHO, PowerTools should be as agnostic as possible, JUL should be used for Logging as it's the Standard in the Java Language.
The Implementer should have the ability to choose the underlying Logging Backend via SLF4J. E.g.: If the Implementer wants to use Logback, he can just bridge all calls using jul-to-slf4j.jar.