CODEWITHSUNDEEP
kibanakibana-4elasticsearch
spuder
spuder

Reputation: 18387

elasticsearch / kibana errors "Data too large, data for [@timestamp] would be larger than limit

On my test ELK cluster, I'm encountering the following error when trying to see data from the last week. 

Data too large, data for [@timestamp] would be larger than limit

The warning about shards failing appears to be misleading because the elasticsearch monitoring tools kopf and head show that all shards are working properly, and the elastic cluster is green.

enter image description here

One user in the google group for elasticsearch suggested increasing ram. I've increased my 3 nodes to 8GB each with a 4.7GB heap, but the issue continues. I'm generating about 5GB to 25GB of data per day, with a 30 day retention.

Upvotes: 33

Views: 44886

Answers (4)

slm
slm

Reputation: 16416

The _cache API also supports targeted clearing to the threadpools for fielddata, request, etc. through just normal URI query parameters.

Prototype, see the placeholder <cache type>:

$ curl -XPOST \
    -H "Content-Type: application/json" \
    'http://localhost:9200/_cache/clear?<cache type>=true'

Examples

$ curl -XPOST \
    -H "Content-Type: application/json" \
    'http://localhost:9200/_cache/clear?fielddata=true'

$ curl -XPOST \
    -H "Content-Type: application/json" \
    'http://localhost:9200/_cache/clear?request=true'

NOTE: You can also target specific indexes, substitute below where <index>:

$ curl -XPOST \
    -H "Content-Type: application/json" \
    'http://localhost:9200/<index>/_cache/clear?request=true'

References

Upvotes: 6

FChiri
FChiri

Reputation: 747

The problem is the memory given to the Elasticsearch by ES_JAVA_OPTS.

Try to give more memory with: ES_JAVA_OPTS="-Xmx2g -Xms2g".

Upvotes: 10

spuder
spuder

Reputation: 18387

Clearing the cache alleviates the symptoms for now.

http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clearcache.html

Clear a single index 

curl -XPOST 'http://localhost:9200/twitter/_cache/clear'

Clear multiple indicies

curl -XPOST 'http://localhost:9200/kimchy,elasticsearch/_cache/clear'

curl -XPOST 'http://localhost:9200/_cache/clear'

Or as suggested by a user in IRC. This one seems to work the best. 

curl -XPOST 'http://localhost:9200/_cache/clear' -d '{ "fielddata": "true" }'

Update: these errors went away as soon as the cluster was moved to a faster hypervisor

Upvotes: 61

Greg Bray
Greg Bray

Reputation: 15697

Clearing the cache didn't work for our cluster. When checking the various nodes using http://x.x.x.x:9200/_cat/indices?v&s=index:desc one was giving the above error and the others had invalid pointer errors. I restarted the elastic service on the one giving the rate limit/data too large error. When it came back online there were some unassigned shards, which I fixed by dropping the replication count to a lower number (only need to do this on one of the nodes to change the index settings for the cluster):

IFS=$'\n'
for line in $(curl -s 'elastic-search.example.com:9200/_cat/shards' | fgrep UNASSIGNED); do
  INDEX=$(echo $line | (awk '{print $1}'))
  echo start $INDEX
  curl -XPUT "elastic-search.example.com:9200/$INDEX/_settings" -d '{
      "index" : {
        "number_of_replicas" : 1
      }
    }
    '
done

# Check shard/cluster status (may take some time to rebalance):
# http://elastic-search.example.com:9200/_cat/shards?v&s=index:desc
# http://elastic-search.example.com:9200/_cluster/health?pretty

Also https://discuss.elastic.co/t/data-too-large/32141 seems to mention it may be an issue with the size of the JVM heap.

Upvotes: 1

Related Questions