CODEWITHSUNDEEP
logstashelastic-stacklogstash-configurationelkamazon-elasticsearch
Pyae Phyoe Shein
Pyae Phyoe Shein

Reputation: 13827

Encountered a retryable error. Will Retry with exponential backoff 413

Logstash keep encountering following error message that logs cannot be sent to AWS ElasticSearch.

[2021-04-28T16:01:28,253][ERROR][logstash.outputs.amazonelasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>413, :url=>"https://search-xxxx.ap-southeast-1.es.amazonaws.com:443/_bulk"}

That's why I always need to restart logstash and cannot configure why it causes that issue. Regarding Logstash documentation I reduce pipeline.batch.size size to 100 but it didn't help. Please let me know how to resolve that issue. Thanks.

pipeline.batch.size: 125
pipeline.batch.delay: 50

Upvotes: 0

Views: 9217

Answers (2)

Pyae Phyoe Shein
Pyae Phyoe Shein

Reputation: 13827

I've fixed issue that we need to adjust as to choose correct ES instance size based on max_content_length.

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html

Upvotes: 0

Badger
Badger

Reputation: 4072

A 413 response is "payload too large". It does not make much sense to retry this, since it will probably recur forever and shut down the flow of events through the pipeline. If there is a proxy or load balancer between logstash and elasticsearch then it is possible that that is returning the error, not elasticsearch, in which case you may need to reconfigure the proxy.

The maximum payload size accepted by amazonelasticsearch will depend on what type of instance you are running on (scroll down to Network Limits). For some instance types it is 10 MB.

In logstash, a batch of events is divided into 20 MB chunks as it is sent to elasticsearch. The 20 MB limit cannot be adjusted (unless you want to edit the source and build your own plugin). However, if there is a single large event it has to be sent in one request, so it is still possible for a request larger than that to be sent.

Since 20 MB is bigger than 10 MB this is going to be a problem if your batch size is over 10 MB. I do not think you have any visibility into the batch size other than the 413 error. You will have to keep reducing the pipeline.batch.size until the error goes away.

Upvotes: 1

Related Questions