ListGroups, DescribeGroups support (for ConsumerOffsetChecker type tool)

[ssikdar1]

Hey I was wondering if there was any way either in the the kafka tools or a script with the confluent kafka python module to:

• get a list of all consumer groups
• read the current offset of all consumers in a group

When trying to run these commands for ./kafka-consumer-groups.sh it looks like they only work for Java consumers only:

$ ./kafka-consumer-groups.sh --new-consumer --bootstrap-server XXXX:6667 --describe --group 729a8c43-fd73-4fe6-b8e0-b0d73096f30f
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

./kafka-consumer-groups.sh --new-consumer --bootstrap-server xxxx:6667 --list
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
logstash
...

Thanks,

Shan

[ewencp]

@ssikdar1 That message is a bit misleading. This should work fine for non-Java clients that use the new consumer protocol. In that error message it says "Java consumer API" to contrast with the old Scala API. It's tough to come up with better wording because the way we call the new APIs the "new consumer" isn't really meaningful to folks that are newer to Kafka.

[ssikdar1]

Awesome thanks!

I'm still curious about the second part of the question, is this also possible with a python script using the confluent-kafka module?

[ewencp]

Not at the moment. In the Java API an public, supported AdminClient was recently added. We would like to add the equivalent to librdkafka and the various wrappers but we don't have a timeline for it yet.

[ZacOlsen]

Is there an update on the progress of being able to list all consumer groups?

[austinnichols101]

The kafka-python library has a function to obtain a list from a single or multiple brokers, but performance is VERY slow with multiple brokers (so slow that it is unusable).

[zmaktouf]

Any news about this ticket?

[messense]

I've implemented AdminClient.list_groups API in #948 , you can give it a try.