Elasticsearch queries slow performance

Question

We have setup elasticsearch cluster with 7 nodes. Each node having configuration like 16G RAM, 8 Core cpu, centos 6.

Elasticsearch Version : 1.3.0
Heap Memory is - 9000m

1 Master (Non data)
1 Capable master (Non data)
5 Data node

Having 10 indices, In which one index having 55 million documents [ 254Gi (508Gi with replica) ] size rest all indices having approx 20k documents.

Every 1 seconds there are 5-10 new documents are indexing.

But problem is search is bit slow. Almost taking average of 2000 ms to 5000 ms. Some queries are in 1 secs.

Mapping:

{
    "my_index": {
        "mappings": {
            "product": {
                "_id": {
                    "path": "product_refer_id"
                },
                "properties": {
                    "product_refer_id": {
                        "type": "string"
                    },
                    "body": {
                        "type": "string"
                    },
                    "cat": {
                        "type": "string"
                    },
                    "cat_score": {
                        "type": "float"
                    },
                    "compliant": {
                        "type": "string"
                    },
                    "created": {
                        "type": "integer"
                    },
                    "facets": {
                        "properties": {
                            "ItemsPerCategoryCount": {
                                "properties": {
                                    "terms": {
                                        "properties": {
                                            "field": {
                                                "type": "string"
                                            },
                                            "size": {
                                                "type": "long"
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    },
                    "fields": {
                        "type": "string"
                    },
                    "from": {
                        "type": "string"
                    }
                    "id": {
                        "type": "string"
                    },
                    "image": {
                        "type": "string"
                    },
                    "lang": {
                        "type": "string"
                    },
                    "main_cat": {
                        "properties": {
                            "Technology": {
                                "type": "double"
                            }
                        }
                    },
                    "md5_product": {
                        "type": "string"
                    },
                    "post_created": {
                        "type": "long"
                    },
                    "query": {
                        "properties": {
                            "bool": {
                                "properties": {
                                    "must": {
                                        "properties": {
                                            "query_string": {
                                                "properties": {
                                                    "default_field": {
                                                        "type": "string"
                                                    },
                                                    "query": {
                                                        "type": "string"
                                                    }
                                                }
                                            },
                                            "range": {
                                                "properties": {
                                                    "main_cat.Technology": {
                                                        "properties": {
                                                            "gte": {
                                                                "type": "string"
                                                            }
                                                        }
                                                    },
                                                    "sub_cat.Technology.computers": {
                                                        "properties": {
                                                            "gte": {
                                                                "type": "string"
                                                            }
                                                        }
                                                    }
                                                }
                                            },
                                            "term": {
                                                "properties": {
                                                    "product.secondary_cat": {
                                                        "type": "string"
                                                    }
                                                }
                                            }
                                        }
                                    }
                                }
                            },
                            "match_all": {
                                "type": "object"
                            }
                        }
                    },
                    "secondary_cat": {
                        "type": "string"
                    },
                    "secondary_cat_score": {
                        "type": "float"
                    },
                    "size": {
                        "type": "long"
                    },
                    "sort": {
                        "properties": {
                            "_uid": {
                                "type": "string"
                            }
                        }
                    },
                    "sub_cat": {
                        "properties": {
                            "Technology": {
                                "properties": {
                                    "audio": {
                                        "type": "double"
                                    },
                                    "computers": {
                                        "type": "double"
                                    },
                                    "gadgets": {
                                        "type": "double"
                                    },
                                    "geekchic": {
                                        "type": "double"
                                    }
                                }
                            }
                        }
                    },
                    "title": {
                        "type": "string"
                    },
                    "product": {
                        "type": "string"
                    }
                }
            }
        }
    }
}

We are using Default Analyzer.
Any Suggestion? Does this configuration is not enough?

Answer 1

To add to Torsten Engelbrecht's answer, default analyzer might be part of the culprit. This analyzer will index every form of each word as a separate token, meaning that a single verb in a language with complex conjugation can be indexed a dozen times. Also, that degrades the quality of the search results. The same applies if your documents contain formatting information (HTML markup ?).

More, stop words are disabled by default, meaning that each "the", "a"... in english for instance will be indexed as well.

You should consider using localized analyzers (snowball analyzer maybe ?) and stop words for the language used in your documents in order to limit the inverted index size and this way, increase performance.

Also, consider making not_analyzed fields as md5, urls, ids, and other sorts of unsearchable fields.

Answer 2

Looks like the indices can not fit into memory, so there will be some more disk I/O going on. Do you use SSDs? If not you should get some.

Besides this your nodes need more resources (memory, CPU) to handle that index size.

I am a little surprised about the sizes here: ~250 GB for "just" 55 million documents is huge and I don't see you are storing any bigger blobs there (I might be mistaken, its hard to see just from the mapping definition). Maybe you can consider to keep some data not analyzed in case you don't need to query it, but just retrieve it. That would reduce the index size.

Except this I have no other ideas, without knowing all the relevant infrastructure in more detail.