Excessive memory usage with destinationIsPattern in Spring Kafka 3.3.5 when matching many topics

[angelalvro]

In what version(s) of Spring for Apache Kafka are you seeing this issue?

• Spring Boot: 3.4.5
• Spring Kafka: 3.3.5
• Kafka Clients: 3.8.1

Describe the bug

After upgrading from Spring Kafka 3.1.1 to 3.3.5, we observed unexpected memory behavior when configuring a consumer using a regular expression that matches a large number of topics and partitions in the Kafka broker.

The application starts consuming memory progressively until it eventually fails with a java.lang.OutOfMemoryError: Java heap space.

This behavior was not present in version 3.1.1 with the same configuration and number of topics.

To Reproduce

• Set up a Spring Boot project with Spring Kafka 3.3.5 and Kafka Clients 3.8.1.
• Configure a consumer with destinationIsPattern: true using a regular expression that matches a large number of topics and partitions (e.g., ^(tp|TP).tracing-topic.(?!.-dlt$).$).
• Consume messages from the topics matching the pattern.
• Monitor the application’s memory usage over an extended period.
• Observe a progressive heap growth leading to a java.lang.OutOfMemoryError: Java heap space.
• Analyze the heap dump to identify retention of multiple instances related to Kafka consumers and containers.

Expected behavior

When using a topic pattern with destinationIsPattern: true, Spring Kafka is expected to efficiently manage the consumption of all matching topics without causing a progressive and uncontrolled increase in memory usage. Consumption should remain stable and heap usage should stay constant, similar to the behavior observed in version 3.1.1.

Sample

https://github.com/angelalvro/kafka-pattern-memory-issue

[artembilan]

The destinationIsPattern is Spring Cloud Stream option. More over the pattern matching functionality is a part of Apache Kafka itself . So, any idea why the report is here in Spring for Apache Kafka?
Thanks

[angelalvro]

The destinationIsPattern is Spring Cloud Stream option. More over the pattern matching functionality is a part of Apache Kafka itself . So, any idea why the report is here in Spring for Apache Kafka? Thanks

Thank you for your response.

You're right that destinationIsPattern is a Spring Cloud Stream property. However, the memory consumption issue we're observing appears to be related to Spring for Apache Kafka, specifically involving the KafkaMessageListenerContainer and its associated consumer thread(s).

After upgrading from Spring Kafka 3.1.1 to 3.3.5 — using the same configuration and number of topics — we noticed a progressive memory increase that eventually leads to a java.lang.OutOfMemoryError. Heap dump analysis shows a high number of retained objects linked to the listener container and Kafka consumer infrastructure.

Since this issue did not occur with Spring Kafka 3.1.1, it seems likely that a change in the internal handling of pattern-based subscriptions between versions may be contributing to this memory behavior. This is why we are reporting it here under Spring for Apache Kafka, as the root cause appears to be within the framework's container lifecycle or memory management.

We're happy to provide more technical details or heap dump insights if needed.

[artembilan]

Sorry, I still don't see how Spring for Apache Kafka causes the problem.
This is the code from Spring Cloud Stream Kafka Binder:

		final ContainerProperties containerProperties = anonymous
				|| groupManagement
						? usingPatterns
								? new ContainerProperties(Pattern.compile(topics[0]))
								: new ContainerProperties(topics)
						: new ContainerProperties(topicPartitionOffsets);

this one is used for:

final ConcurrentMessageListenerContainer<K, V> messageListenerContainer =
			new ConcurrentMessageListenerContainer<>(consumerFactory, containerProperties) {

That one is used like this here in Spring for Apache Kafka (the mentioned KafkaMessageListenerContainer):

				if (topicPattern != null) {
					subscribingConsumer.subscribe(topicPattern, rebalanceListener);
				}
				else {
					subscribingConsumer.subscribe(Arrays.asList(this.containerProperties.getTopics()), // NOSONAR
							rebalanceListener);
				}

The rest is up to Apache Kafka client itself.
Yes, there was a jump in org.apache.kafka:kafka-clients version from 3.6.2 in spring-kafka.3.1.x to 3.8.x in spring-kafka.3.3.x.

So, it might turn out the problem with memory is definitely in the KafkaConsumer itself.

Any chances to minimize your sample application to at least spring-kafka with Spring Boot auto-configuration?
I mean without Spring Cloud Stream to narrow the scope.
Of course, the next step would be from me to ask you to minimize the scope down to kafka-clients itself.
Since there was no any logic in Spring for Apache Kafka to deal with those patterns some way outside of the KafkaConsumer expectations.

[sobychacko]

@angelalvro In addition to what Artem suggested above, it maybe worth to reach out to the apache kafka community (kafka jira or their mailing list) to see if there is any known issues with that kafka-client version. Since you haven't updated Spring Cloud Stream, it is unlikely that it is causing the issue. In Spring Kafka itself, that part of the code hasn't been changed much in those versions you mentioned.

[angelalvro]

Regarding the memory issues I previously reported, I have conducted a new heap dump analysis and now have clearer evidence that could help identify the root cause of the problem — possibly related to memory accumulation in thread-local variables within a context involving Spring Kafka, Spring Cloud Stream, or Apache Kafka.

Below are the most relevant findings (see attached images):

• Image 1: The analysis shows that a single thread (XNIO-1 task-3) is holding approximately 60% of the heap memory, due to a local variable containing a very large String[] array.
  

• Image 2: This array contains millions of entries, suggesting a continuous accumulation that may not be properly released.
  

• Image 3: Examining the content of some entries reveals integer values represented as strings, which could be related to partition IDs or offsets.
  

Our hypothesis is that some component is using a thread-local structure to store information related to offsets or processing state, and under certain load conditions, this structure is not properly cleared, leading to a memory leak.

We cannot yet confirm whether this behavior originates directly from Spring Kafka, Spring Cloud Stream, or the base Kafka client.

We hope this information helps focus the analysis better and determine whether this is expected behavior, a potential misuse, or a memory management issue in one of the involved libraries.

[artembilan]

Sorry, but I don't see the Kafka word anywhere on your images.
Those are so generic, that it could be anything from your application.
I don't see yet how we can help.