Conda packages are available on conda-forge

[mrocklin]

Just an FYI: Conda packages for confluent-kafka-python and librdkafka are available on conda-forge, a community-run build system for binary packages that is common among Python's data science community. This allows Python users on Linux or OSX using Python 2 or 3 to download and install both packages with the following command:

conda-install -c conda-forge python-confluent-kafka

In case readers are unaware, conda-forge is used at essentially every Python tutorial at the Python data-science oriented conferences (pydata, scipy, strata, etc.) and now often serves as the source of recipes for the commercial Anaconda distribution.

Three thoughts

1. It might be useful for someone at Confluent to check over the build recipes to ensure that everything is as it should be.
2. It might be useful to ensure that someone from Confluent has rights to update these package on the conda-forge build service in the future
3. It might be useful to include this as an option within installation documentation

Build recipes

• https://github.com/conda-forge/librdkafka-feedstock/blob/master/recipe/meta.yaml
• https://github.com/conda-forge/librdkafka-feedstock/tree/master/recipe

This is just an FYI, not a bug report. Feel free to close.

[edenhill]

This is absolutely great!

Thank you

[edenhill]

The recipe references libgsasl, which is not used by librdkafka.
Librdkafka (optionally) depends on cyrus-sasl (aka libsasl2) to provide SASL GSSAPI/Kerberos support on Linux and OSX (comes with the system install on OSX).

[est]

@edenhill how to add libsasl2 support to conda's librdkafka? I installed them both and they didn't seem to work together.

KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create consumer: No provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM"}

conda-forge/python-confluent-kafka-feedstock#18

[edenhill]

@est the conda packages are not maintained by confluent, maybe @mrocklin can help?

[mrocklin]

I am also not maintaining those packages full time. It looks like @est also raised an issue on the conda forge issue tracker a few minutes ago. Let's see if some of the conda forge maintainers pick this up.

The best solution, in my opinion, is for @est to be able to solve this problem and contribute the patch to conda-forge. However it looks like @est might need a bit of guidance to accomplish this. I personally don't know much about this space, so am unlikely to be of much help.

[est]

Thanks for the info guys, I am struggling with a CentOS machine without root privileges (company rules). Getting a working gcc is difficult enough for me. conda is my best hope as it provides binaries directly.

[mrocklin]

conda install gcc?

[edenhill]

cyrus-sasl and its dependencies are, unfortunately, bit of a mess to install without the system's default package manager. Best of luck :)

[mrocklin]

It looks like they too are conda installable :)

$ conda search cyrus-sasl
Loading channels: done
# Name                       Version           Build  Channel
cyrus-sasl                    2.1.26      h26aff7b_2  pkgs/main
cyrus-sasl                    2.1.26      h26aff7b_3  pkgs/main
cyrus-sasl                    2.1.26      ha054001_1  pkgs/main
cyrus-sasl                    2.1.26      hb48c43a_4  pkgs/main

[est]

@mrocklin I ran conda install cyrus-sasl librdkafka confluent-kafka-python and ran my .py script it still complains recompile librdkafka with libsasl2

[mrocklin]

Yeah, I'm not saying that installing them both means that they're compiled together. I'm just reponding to @edenhill 's comment above that it's hard to get cyrus-sasl without a system package manager.

[est]

ok. Thanks for the troubles.

Later I found that CentOS 7's librdkafka-devel still has no 1.0.1 rpm. I am on my adventure to look for it.

[est]

conda install -c stuarteberg -c conda-forge librdkafka=1.0.1=hf484d3e_1 now has sasl.kerberos support.

conda-forge/librdkafka-feedstock#34