Kafka Streams tracing no longer working after upgrading to Spring Boot 3 + Micrometer Tracing

[j-tim]

Context

At the moment I'm migrating some Spring Boot 2.7.x, Spring for Apache Kafka applications to Spring Boot 3.

I followed the migration guide and Spring Kafka documentation regarding migrating from Spring Cloud Sleuth to Micrometer Tracing

• https://docs.spring.io/spring-kafka/reference/html/#observation-gen
• https://github.com/spring-cloud/spring-cloud-release/wiki/Spring-Cloud-2022.0-Release-Notes
• https://github.com/micrometer-metrics/tracing/wiki/Spring-Cloud-Sleuth-3.1-Migration-Guide

Distributed tracing works fine for:

• producing to Kafka (by enabling observation on the KafkaTemplate)
• and consuming from Kafka (by enabling observation via the container properties of the KafkaListenerContainerFactory)

But for Kafka streams application (using the KafkaStreamBrancher in my minimal, reproducible, sample) the tracing is no longer working after the upgrade.

In what version(s) of Spring for Apache Kafka are you seeing this issue?

Spring for Apache Kafka: 3.0.4

Migrating from:

• Spring Boot 2.7.9
  • Spring Cloud: 2021.0.6
  • Spring for Apache Kafka: 2.8.11
  • Spring Cloud Sleuth

to:

• Spring Boot 3.0.4
  • Spring Cloud: 2022.0.1
  • Spring for Apache Kafka: 3.0.4
  • Micrometer Tracing (Open Telemetry + Zipkin)

Describe the bug

For the Kafka streams application the tracing is no longer working after the upgrade.

Spring Boot 2.7.9, Spring Kafka and Spring Cloud Sleuth:

Spring Boot 3.0.4, Spring Kafka and Micrometer Tracing (OTEL)

Expected behavior

I expect the traces of my Kafka Streams application are reported in this case to Zipkin.

Sample to reproduce the issue

I put some effort in creating a GitHub repository with a minimal, reproducible, sample to reproduce the issue. The README of the sample project contains all the step to reproduce the issue.

Branches in the example repository:

• main: branch that contains the example to reproduce the issue with Spring Boot 3.0.4, Spring Kafka and Micrometer Tracing
• spring-boot-2.7-spring-cloud-sleuth: branch that contains a fully working example based on Spring Boot 2.7.9 and Spring Cloud Sleuth

Did I overlook something in the documentation to enable tracing for my Kafka Streams application? Or is tracing for Kafka Stream applications not supported yet using Spring for Apache Kafka and Micrometer Tracing?

[garyrussell]

Spring is not involved with instrumenting Kafka Streams at all - Spring is only used to set up the stream; there is no runtime involvement.

My (limited) understanding is that Sleuth uses brave to instrument the streams producer/consumer.

I don't know if it's practical for this project to replace the sleuth functionality for Streams - I'll have to defer that to the Micrometer team.

cc/ @marcingrzejszczak

[marcingrzejszczak]

If there's an option for Spring Kafka to instrument Kafka Streams by any means then that would be great (although I don't know if that's feasible?).

[garyrussell]

How does Sleuth do it?

[marcingrzejszczak]

As you said - it does it via Brave's components (https://github.com/spring-cloud/spring-cloud-sleuth/blob/c795675cf93c666c3c13f9bbb9eadbde1f1aa775/spring-cloud-sleuth-autoconfigure/src/main/java/org/springframework/cloud/sleuth/autoconfig/brave/instrument/messaging/BraveKafkaStreamsAutoConfiguration.java).

Here you can see how they exactly do it https://github.com/openzipkin/brave/tree/master/instrumentation/kafka-streams

[garyrussell]

I think all we can do here is explain how to set the KafkaClientSupplier on the StreamsBuilderFactoryBean, as is done by Sleuth here:

https://github.com/spring-cloud/spring-cloud-sleuth/blob/c795675cf93c666c3c13f9bbb9eadbde1f1aa775/spring-cloud-sleuth-autoconfigure/src/main/java/org/springframework/cloud/sleuth/autoconfig/brave/instrument/messaging/BraveKafkaStreamsAutoConfiguration.java#L106

[marcingrzejszczak]

Not really cause that's just delegating to what brave does. What if someone uses open telemetry or another tracer?

[garyrussell]

What does Sleuth do in those cases?

The only extension point that Kafka Streams provides is the client supplier; so each library needs to provide its own implementation to decorate the producer/consumer; there is nothing else that this project can do.

[marcingrzejszczak]

Sleuth only supports the dedicated Brave instrumentation.

[j-tim]

Thanks, Gary and Marcin for your responses.
I understand just delegating to brave is not the way to go.

Based on your suggestions I try to implement a custom Kafka client supplier.
To set the custom KafkaClientSupplier on the StreamsBuilderFactoryBean:

@Bean
public StreamsBuilderFactoryBeanCustomizer streamsBuilderFactoryBeanCustomizer() {
   return factoryBean -> factoryBean.setClientSupplier(new TracingKafkaClientSupplier());
}

The TracingKafkaClientSupplier extends the DefaultKafkaClientSupplier.

Can you give me some hints on how to implement this custom supplier?
Should this custom supplier use the ObservationRegistry of Micrometer? Or should I directly use the OpenTelemetry API
like in this example: https://github.com/ppatierno/kafka-opentelemetry/blob/main/kafka-streams-wrap/src/main/java/io/ppatierno/kafka/opentelemetry/Streams.java#L42

Would it not make more sense if Spring or Micrometer will provide a 'Tracing Kafka client supplier' that will be auto-configured or registered when observation is enabled like how it's done for the KafkaTemplate and KafkaListenerContainer?