CODEWITHSUNDEEP
httpelasticsearchluceneapache-nifi
Dinosaurius
Dinosaurius

Reputation: 8628

How to execute ElasticSearch agg query from Apache Nifi using InvokeHTTP?

I want to execute the following query from Apache Nifi:

GET /myindex/_search
{
  "size": 0,
  "aggs": {
    "range": {
        "date_range": {
            "field": "Datetime",
            "ranges": [
                { 
                   "from": "2017-02-17T16:00:00Z||-1H/H",
                   "to": "2017-02-17T16:00:00Z||/H" 
                }
            ]
        }
    }
  }
}

I want to get the value of doc_count.

I tried to use InvokeHTTP and directly pasted the above-defined query into the field HTTP Method. I also set Remote URL to http://localhost:9200. I connected InvokeHTTP with PutFile in order to save a response on the disk. The relationship is Response.

When I run InvokeHTTP, it does not give me any error. However, it neither outputs any result (FlowFile for Response). I am sure that the result is not an empty set, because I tested this query with curl.

What is wrong with my approach? Should I define HTTP Method in some different way?

Upvotes: 0

Views: 2637

Answers (2)

Simon Lang
Simon Lang

Reputation: 592

There is a JsonQueryElasticsearch processor, which supports JSON elasticsearch queries.

JsonQueryElasticsearch configuration

Your query goes into the Query field. The processor has hits and aggregations relations for you to process the result.

Upvotes: 0

mattyb
mattyb

Reputation: 12093

For verbs like POST, PUT, and PATCH, you'd normally have the above JSON body as the content of a flow file and pass that to InvokeHttp, setting the correct verb and URL there. However the InvokeHttp documentation states that the message body will not be sent for a GET verb.

The good news is that the Elasticsearch REST API for the search endpoint supports both GET and POST. From their latest doc: "Both HTTP GET and HTTP POST can be used to execute search with body. Since not all clients support GET with body, POST is allowed as well."

I would set the content of a flow file to the above JSON body (perhaps using GenerateFlowFile or ReplaceText), then use POST as the verb.

Upvotes: 1

Related Questions