CODEWITHSUNDEEP
apache-kafkafilebeat
parttime-penzi
parttime-penzi

Reputation: 11

filebeat-kafka:WARN producer/broker/0 maximum request accumulated, waiting for space

when filebeat output data to kafka , there are many warning message in filebeat log.

..

*WARN producer/broker/0 maximum request accumulated, waiting for space

*WARN producer/broker/0 maximum request accumulated, waiting for space

..

nothing special in my filebeat config:

..

output.kafka:

  hosts: ["localhost:9092"]

  topic: "log-oneday"

..

i have also updated these socket setting in kafka:

...

socket.send.buffer.bytes=10240000

socket.receive.buffer.bytes=10240000

socket.request.max.bytes=1048576000

queued.max.requests=1000

...

but it did not work.

is there something i missing? or i have to increase those number bigger?

besides , no error or exception found in kafka server log

is there any expert have any idea about this ?

thanks

Upvotes: 1

Views: 4694

Answers (2)

Thiago Falcao
Thiago Falcao

Reputation: 5043

You need to configure 3 things:

  • Brokers

  • Filebeat kafka output

  • Consumer

Here a example (change paths according your environment).

Broker configuration:

# open kafka server configuration file
vim /opt/kafka/config/server.properties

# add this line
# The largest record batch size allowed by Kafka.
message.max.bytes=100000000

# restart kafka service
systemctl restart kafka.service

Filebeat kafka output:

output.kafka:
  ...
  max_message_bytes: 100000000

Consumer configuration:

# larger than the max.message.size
max.partition.fetch.bytes=200000000

Upvotes: 1

dvshulgin
dvshulgin

Reputation: 21

Apparently you have only one partition in your topic. Try to increase partitions for the topic. See the links below for more information.

More Partitions Lead to Higher Throughput

Try the following command (replacing info with your particular use case):

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name  --partitions 40

Upvotes: 2

Related Questions