CODEWITHSUNDEEP
elasticsearchkibana
Narendra522
Narendra522

Reputation: 167

How to search for indices which does not have a specific field in Elasticsearch

I have an elastic search cluster which has 1000+ indices, most of my indices has specific time stamp in epoch format, however few indices has time stamp in UTC and that is causing query crash when some one is trying to search from kibana, I am trying to list all the indices which does not have a filed called ts, I am able get field mapping by doing curl culr -XGET http://es-node1:9200/*/_mapping/field/ts?pretty

however going through all the 1000+ results are very tedious, could some one please point me to right query to exclude all the indices which does not have field "ts"

If I run the query from kibana nodes in my cluster starts spiking the cpu, the filter I am running is 

"filter": {
  "list": {
    "0": {
      "type": "time",
      "field": "ts",
      "from": "now-5m",
      "to": "now",
      "mandate": "must",
      "active": true,
      "alias": "",
      "id": 0
    },
    "1": {
      "type": "querystring",
      "query": "_exists_:ts",
      "mandate": "must",
      "active": true,
      "alias": "",
      "id": 1
    }
  },
  "ids": [
    0,
    1
  ],
  "hide": true
}

},

and this is the log I am getting in my ES node.

[2017-07-19 06:35:12,874][DEBUG][action.search            ] [Gargoyle] [ws][4], node[FbXhw2ERQT6QDPmNCuOMGQ], [R], v[21], s[STARTED], a[id=iH_6483qRHmJGSk95rOtQg]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4078c587] lastShard [true]

RemoteTransportException[[Captain America][es-node1:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":""}}]}},"filter":{"bool":{"must":[{"range":{"ts":{"from":1500445811001,"to":1500446111001}}},{"fquery":{"query":{"query_string":{"query":"ts:(1500446010000)"}},"_cache":true}}]}}}},"highlight":{"fields":{},"fragment_size":2147483647,"pre_tags":["@start-highlight@"],"post_tags":["@end-highlight@"]},"size":1000,"sort":[{"_score":{"order":"desc"}},{"ts":{"order":"desc"}}]}]]; nested: SearchParseException[No mapping found for [ts] in order to sort on]; Caused by: SearchParseException[failed to parse search source [{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":""}}]}},"filter":{"bool":{"must":[{"range":{"ts":{"from":1500445811001,"to":1500446111001}}},{"fquery":{"query":{"query_string":{"query":"ts:(1500446010000)"}},"_cache":true}}]}}}},"highlight":{"fields":{},"fragment_size":2147483647,"pre_tags":["@start-highlight@"],"post_tags":["@end-highlight@"]},"size":1000,"sort":[{"_score":{"order":"desc"}},{"ts":{"order":"desc"}}]}]]; nested: SearchParseException[No mapping found for [ts] in order to sort on]; at org.elasticsearch.search.SearchService.parseSource(SearchService.java:855) at org.elasticsearch.search.SearchService.createContext(SearchService.java:654) at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:620) at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:371) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365) at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: SearchParseException[No mapping found for [ts] in order to sort on] at org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:212) at org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:186) at org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:84) at org.elasticsearch.search.SearchService.parseSource(SearchService.java:838) ... 12 more

I am trying to delete the indices which does not have field "ts" in it, for that I am looking for a query where I can get only the indices which does not have the field ts, Could some one please help me with the query?

Upvotes: 0

Views: 495

Answers (1)

Val
Val

Reputation: 217514

In Kibana, you can use the _exists_ query and add the following to your query

_exists_:ts AND ...the rest of your query

Upvotes: 0

Related Questions