Installing librdkafka on Windows to support Python development

Question

A little background: I am working on some python modules that other developers on our team will use. A common theme of each module is that one or more messages will be published to Kafka. We intend at this time to use the Confluent Kafka client. We are pretty new to python development in our organization -- we have traditionally been a .NET shop.

The complication: while the code that we create will run on Linux (rhel 7), most of the developers will do their work on Windows.

So we need the librdkafka C library compiled on each developer machine (which has dependencies of its own, one of which is OpenSSL). Then a pip install of confluent-kafka should just work, which means a pip install of our package will work. Theoretically.

To start I did the install on my Linux laptop (Arch). I knew I already had OpenSSL and the other zip lib dependencies available, so this process was painless:

• git clone librdkafka repo
• configure, make and install per the README
• pip install confluent-kafka
• done

The install of librdkafka went into /usr/local:

/usr/local/lib/librdkafka.a
/usr/local/lib/librdkafka++.a
/usr/local/lib/librdkafka.so -> librdkafka.so.l
/usr/local/lib/librdkafka++.so -> librdkafka++.so.l
/usr/local/lib/librdkafka.so.l
/usr/local/lib/librdkafka++.so.l
/usr/local/lib/pkgconfig/rdkafka.pc
/usr/local/lib/pkgconfig/rdkafka++.pc
/usr/local/include/librdkafka/rdkafkacpp.h
/usr/local/include/librdkafka/rdkafka.h

Now the painful part, making it work on Windows:

• install precompiled OpenSSL
• git clone librdkafka repo
• open in VS2015
• install libz via NuGet
• build solution
• install to where???

This is where I'm stuck. What would a standard install on a Windows 7/8/10 machine look like?

I have the following from the build output, but no idea what should go where in order to make the pip install confluent-kafka "just work":

/librdkafka/win32/Release/librdkafka.dll
/librdkafka/win32/Release/librdkafka.exp
/librdkafka/win32/Release/librdkafka.lib
/librdkafka/win32/Release/librdkafkacpp.dll
/librdkafka/win32/Release/librdkafkacpp.exp
/librdkafka/win32/Release/librdkafkacpp.lib
/librdkafka/win32/Release/zlib.dll
<and the .h files back in the src>

Any recommendations on an install location?

Answer 1

I assume you have successfully followed instructions from MichaelZhangCA (https://github.com/MichaelZhangCA/confluent-kafka-python/) from previous post.

If you did so, these probably were the last two commands executed:

::Install confluent-kafka
cd C:\confluent-kafka-python\confluent-kafka-python-0.11.4
python setup.py install

If that is correct, those DLLs were created under C:\confluent-kafka-python\librdkafka-reference\release\.

All you have to do is to copy them to a diretory already in Widnows' PATH.

For example, I use Anaconda 5.2 For Windows, with python 3.6. My Anaconda Prompt has an empty directory in PATH, so I copied there those DLL's:

::Anaconda Prompt - copy DLLs to a directory already in PATH
mkdir %CONDA_PREFIX%\Library\usr\bin
copy C:\confluent-kafka-python\librdkafka-reference\release %CONDA_PREFIX%\Library\usr\bin

If you don't use Anaconda, just copy those DLL's to any other directory in Windows' PATH. You may also leave them in C:\confluent-kafka-python\librdkafka-reference\release, and add this directory to PATH.

Answer 2

It's an old question but seems still no easy answer yet. Also Confluent seems too busy to work on Windows supporting...

I had the same headache couple weeks ago and after some research, I managed to make it work for me on Windows. I logged my finding and uploaded a pre-compiled library to my Git, please check and see if it helps. :D

https://github.com/MichaelZhangCA/confluent-kafka-python

My environment is Python 3.6 64 bit version but ideally, it should also work for 32 bit if you follow the same approach.

Answer 3

I'm not sure where the ideal place to install on Windows would be, but I ran the following test with some success.

I copied my output and headers to C:\test\lib and C:\test\include, then ran a pip install with the following options:

 pip install --global-option=build_ext --global-option="-LC:\test\lib" --global-option="-IC:\test\include" confluent-kafka

Unfortunately, this doesn't quite work because the confluent-kafka setup does not support Windows at this time: https://github.com/confluentinc/confluent-kafka-python/issues/52#issuecomment-252098462