Observability : Kafka Consumer/Producer metrics and tracing.

[surenw-serenade]

Describe the solution you would like

Is there a recommended approach to instrumenting the kafka-go library with OpenTelemetry?

Standard consumer and producer metrics with support for standard prometheus or otel export capability would be useful.

Supporting documentation

https://kafka.apache.org/20/generated/producer_metrics.html
https://kafka.apache.org/20/generated/consumer_metrics.html

[Sovietaced]

It looks like client side metrics would ideally be implemented with the interceptor concept (which I don't think has been added to this library yet): https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors

[mfelipe]

My perspective using Prometheus: you can call Writer.Stats() to get a WriterStats object full of metrics and do something with that. The same is valid for Reader.

I'm using reflection to automatically initialize the collectors (only counters) and to gather (Add) the values to each collector. Values are updated when /metrics is called. The only downside for me is that to get metrics by topic you need readers and writers for each topic.

[fastbean-au]

+1 for this.