Support structured logging as alternative to pattern layout logging

[mp911de]

Spring Boot offers logging using a pattern for various log frameworks. In scenarios, where more context details would be useful, users need to adopt the pattern. Each value from the MDC needs to be configured independently which is a consistent source of work. It would be great to opt-in for a JSON log format that is used instead of the pattern layout to have a flexible field schema for later processing within log aggregators such as Splunk, Graylog, or Logstash.

Use-cases

• Using MDC to store contextual details helps while tracing and debugging applications. Currently, the pattern layout is configured in a static way (in config files) while MDC values can be added dynamically. Adding dynamically values to the JSON structure from the MDC does not necessarily require additional configuration each time a new MDC entry is added to the application. That's actually the use case from which the power comes.
• Log forwarders need to be adopted for each change of the log format. Adding Spring Clouds Sleuth to use the slf4j span logger requires for instance adoption of the log format.
• Multiline logging: Configuring log forwarders for parsing multiline logs is challenging. Exceptions can be nicely wrapped within a JSON field; The syntax of JSON makes it easy because no additional multiline configuration needs to be set up
• JSON logs can be either picked up from stdout by an online-forwarder or written into a file that is read by logstash forwarder (or other tools)

See https://github.com/mp911de/logstash-gelf for examples.

[cemo]

This would be a nice addition. I had to customize https://github.com/logstash/logstash-logback-encoder project to integrate with Logentries which is also supporting JSON logs.

Please note that logstash-logback-encoder is general-purpose JSON logging mechanism project.

[philwebb]

@mp911de Is there anything that's currently preventing JSON logging from working for you? I'm not sure that we really want to take on too much more responsibility in this area. We already have quite a maintenance burden trying to support three different logging systems.

[mp911de]

There's nothing which prevents JSON logging/log forwarding. It's more about the barrier to configure logging and set up conditional JSON logging. Usually, in dev, one does not want to have JSON logs but would enable JSON logging in staging/production. Using different logging config files and specifying logging.config=(my config file) per environment increases the complexity.

I fully understand the maintenance efforts from your side. I'm considering maybe another approach with building/providing a starter in my own repo as I did JSON logging earlier. This way an extension existed one could use, and you don't have the maintenance hassle.

[philwebb]

That would certainly be better from our point of view. We could always add other extension points if needed.

[mp911de]

I'll follow that path then, thanks for your feedback.

[imjorge]

@mp911de have you gotten to implement the spring boot starter you mentioned?

[mp911de]

@imjorge I haven't been doing much on that end.