CODEWITHSUNDEEP
dockerapache-kafka
August
August

Reputation: 1819

Apache kafka invalid receive size

I am running the following docker compose:

version: '3'
services:
    zookeeper:
        image: wurstmeister/zookeeper
        ports:
            - '2181:2181'
    kafka:
        image: wurstmeister/kafka:2.12-2.3.0
        ports:
          - "9094:9094"
        environment:
            KAFKA_ADVERTISED_HOST_NAME: $HOST
            KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
            KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
            KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://$HOST:9094
            KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
            KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
            # KAFKA_SOCKET_REQUEST_MAX_BYTES: 1347375956
        volumes:
            - 'kafka_vol:/var/run/docker.sock'
    ui:
        image: landoop/kafka-topics-ui
        ports:
            - "8000:8000"
        environment:
            KAFKA_REST_PROXY_URL: http://kafka:9092
            PROXY: "true"
            MAX_BYTES: 50000
            LAZY_LOAD_TOPIC_META: "true"
            PROXY_SKIP_VERIFY: "true"
volumes:
    kafka_vol: {}

Server:

The server is 8GB RAM, when looking at docker stats, kafka and zookeeper barely exceed 1G combined.

Problem:

After doing a clean install I've trie to open landoop/kafka-topics-ui in my browser, however requests failed and I've noticed that in docker compose logs the following error was shown:

[2019-07-22 07:41:29,540] org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)

The same error occurred when I went inside the container for kafka and ran curl http://localhost:9092/topics. After doing a clean install I've trie to open landoop/kafka-topics-ui in my browser, however requests failed and I've noticed that in docker compose logs the following error was shown:

[2019-07-22 07:41:29,540] org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)

The same error occurred when I went inside the container for kafka and ran curl http://localhost:9092. What's causing this issue ?

Upvotes: 6

Views: 14309

Answers (2)

Mickael Maison
Mickael Maison

Reputation: 26895

You cannot directly access Kafka via HTTP. The error you get is because you sent an HTTP request to Kafka. The size in the error message 1195725856 is the 4 first bytes of the HTTP request!

1195725856 is a recognizable value, it's GET decoded as a integer:

>>> struct.unpack("!I", struct.pack("!4s", "GET ".encode("UTF8")))
(1195725856,)

In order to use landoop/kafka-topics-ui, you need to install confluentinc/kafka-rest proxy which provides an HTTP endpoint in front of Kafka. This is mentioned in the README:

Browse Kafka topics and understand what's happening on your cluster. Find topics / view topic metadata / browse topic data (kafka messages) / view topic configuration / download data. This is a web tool for the confluentinc/kafka-rest proxy.

Upvotes: 9

Nishu Tayal
Nishu Tayal

Reputation: 20860

This means, the receiving message size is more than default message size ~ 100 MB which the socket server can accept.

You need to change the below configurations in server.properties to something larger which you want to use(eg. 300 MB) : 

socket.request.max.bytes=300000000

Upvotes: -1

Related Questions