CODEWITHSUNDEEP
dockerapache-kafkakafka-python
Hethcox
Hethcox

Reputation: 108

KafkaConsumer hangs on poll

I have a kafka consumer client in python that runs fine in the terminal (macOS). I deployed the client code to Docker. Now when the client runs it seems to connect to kafka OK, but hangs when I poll for data. In this code the call to consumer.poll() never returns.

def process_video_stream():
global recv_img, first_frame
while keep_polling:
    app.logger.info("polling")
    msg_pack = consumer.poll(timeout_ms=500)
    app.logger.info("back")

    for tp, messages in msg_pack.items():
        for message in messages:
            # message value and key are raw bytes -- decode if necessary!
            # e.g., for unicode: `message.value.decode('utf-8')`
            buf = numpy.array(message.value)
            jpg_as_np = numpy.frombuffer(buf, dtype=numpy.uint8)
            # mat = cv2.UMat(buf)
            img = cv2.imdecode(jpg_as_np, cv2.IMREAD_COLOR)
            if first_frame:
                open_vid_file(img)
                first_frame = False
            writer.write(img)

            app.logger.debug("%s:%d:%d" % (tp.topic, tp.partition,
                                message.offset))

From inside the container I was able to telnet to the kafka server so I don't think it's a networking issue. All the SO answers I can find pertain to people running kafka in the container, not the producers/consumers.

Upvotes: 2

Views: 1357

Answers (1)

Hethcox
Hethcox

Reputation: 108

It turns out that I had the same problem as is common when kafka is in the container. The advertised.listeners needs to be set to the hostname/IP of the host machine, not the default.

Upvotes: 1

Related Questions