CODEWITHSUNDEEP
javaelasticsearchelasticsearch-5
mvreijn
mvreijn

Reputation: 2942

Elastic Java client returns different result from HTTP API

I have an application that is using the Elasticsearch Java API (5.6.7) to execute an aggregation query (terms). I have created the following search document using curl and the HTTP API (relevant information displayed): 

{
  "from" : 0,
  "size" : 0,
  "sort" : [
    {
      "@timestamp" : {
        "order" : "desc"
      }
    }
  ],
  "aggregations" : {
    "level" : {
      "terms" : {
        "field" : "level.keyword",
        "size" : 10,
        "min_doc_count" : 1,
        "shard_min_doc_count" : 0,
        "show_term_doc_count_error" : false,
        "order" : [
          {
            "_count" : "desc"
          },
          {
            "_term" : "asc"
          }
        ]
      }
    }
  }
}

Now that the query has been implemented in my Java program, I notice that the results are different from the HTTP API results!

Both return the exact same meta-information regarding shards, number of hits, etc: 

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 3659,
    "max_score": 0.0,
    "hits": [

    ]
  }

However the returned aggregation from the Java API does not contain any buckets: 

  "aggregations": {
    "level": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [

      ]
    }

while the same aggregation from the HTTP API does contain the buckets:

  "aggregations": {
    "level": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "INFO",
          "doc_count": 2691
        },
        {
          "key": "WARN",
          "doc_count": 776
        },
        {
          "key": "ERROR",
          "doc_count": 192
        }
      ]
    }

I am 100% sure that the search document is the same (copied it from the Java app log).

Q: What can cause this difference?

EDIT My java code for building the query is (contains a lot of references to other classes):

    // Start building the search itself
    SearchRequestBuilder srch = client.prepareSearch(indices.toArray(new String[indices.size()]))
            .setTypes(types.toArray(new String[types.size()])).setFrom(0).setSize(0);
    // Conditional sort order
    if (t.getOrder() != null)
        srch.addSort(t.getOrder().getBuilder());
    // Add aggregationbuilders to this search
    for (NivoStatistic stat : t.getStatistics())
    {
        logger.log(Level.FINER, "Adding statistic {0}", stat.getName());
        srch.addAggregation(stat.getContent());
    }
    // Use a search template    
    NivoQuery qry = t.getQuery();
    SearchTemplateRequestBuilder srchTemplate = new SearchTemplateRequestBuilder(client)
            .setRequest(srch.request())
            .setScript(qry.getTemplateString())
            .setScriptType(ScriptType.INLINE)
            .setScriptParams(qry.getParameterValues())
            ;
    logger.log(Level.FINER, "Prepared search: {0}", srchTemplate.request().getRequest().toString());

The output from the last log statement is what I use to POST via curl -XPOST http://localhost:9200/...

Then execution is done via 

    // Execute the search
    try
    {
        SearchResponse resp = srchTemplate.get().getResponse();
        logger.log(Level.FINER, "Search returned: {0}", resp.toString());
        if (resp.status() == RestStatus.OK && resp.getAggregations() != null)
        {
            for (Aggregation agg : resp.getAggregations().asList())
            {
                // Update response
                t.getResponse().addStat(new NivoStatsHit(agg));
            }
        }
    }
    catch (ElasticsearchException e)
    {
        throw new ApiException(ApiExceptionCode.SEARCH_10061, "Database error: " + e.getDetailedMessage());
    }

Upvotes: 5

Views: 1183

Answers (1)

Andrei Stefan
Andrei Stefan

Reputation: 52368

I started testing your code with Elasticserach 5.6.3 and, while at first it looked doable, then I realized is not that easy. And all seems to boil down to search templates usage.

The main thing with your code is that you are using search templates combined with aggregations. In my tests, also, not even size and from worked ;-). Not sure how it worked for you though. Or maybe you haven't realized that the result is also getting you back the documents themselves, as it's not relevant from your post. The query you are logging looks right, indeed, but the results show that aggregations, size and from are ignored.

So, at this point I started looking into why search templates and aggregations don't seem to work (fyi, resp.getAggregations() returns null). And I found this - https://github.com/elastic/elasticsearch/issues/22766.

I tried to combine a search template builder with a normal search request, but I failed.

Upvotes: 3

Related Questions