CODEWITHSUNDEEP
elasticsearch
ykoo
ykoo

Reputation: 249

how to store elasticsearch query result to new index directly?

I have three questions.

First, I wonder if I can save the mtermvectors query results back to a new index using a single query.

Second, If first question is okay, I wonder if I can change the current format to the nested data type when saving the result.

Last, I wonder if I can do aggregation at the same time. Is it possible?

Now, my code is here.

<pre>
GET /_mtermvectors?pretty=true 
{   
  "docs":[
  {   "_index": "test",    
    "_type": "type1",    
    "_id": "2",    
    "fields":["contents"],    
    "offsets":false,    
    "payloads":false,    
    "positions":false,    
    "term_statistics": true,
    "field_statistics":true,   
    "filter":{
    "min_term_freq":1,
    "max_doc_freq": 3
}   },   
 {   "_index": "test",    
   "_type": "type1",    
   "_id": "1",    
   "fields":["contents"],    
   "offsets":false,    
   "payloads":false,    
   "positions":false,    
   "term_statistics": true,
   "field_statistics":true,  
   "filter":{
  "min_term_freq":1,
  "max_doc_freq": 3
}   
} ] }


PUT /new_index
{
  "mappings": {
    "word": {
    "properties": {
       "termsarray": {
         "type": "nested",
          "properties": {
         "form": { 
           "type":"text",
           "fielddata": true
    }
        }
      }
   }
   }
}

}

GET new_index/_search
{
 "aggs" : {
    "termsarray" : {
        "nested" : {
            "path" : "termsarray"
        },
       "aggs" : {
    "word" : {
       "terms" : { 
         "field" : "termsarray.form",
          "order" : { "sum_score" : "desc" }
        },
        "aggs" : {
            "sum_score" : { "sum" : { "field" : "termsarray.score" }}
        }
    }
 }
     }
 }

}

Upvotes: 5

Views: 5970

Answers (3)

jetnet
jetnet

Reputation: 661

Another simple method to search and store is to use elasticdump with the parameter --searchBody.

Or, if you got your search results from e.g. Kibana like this:

GET your-index/_search?pretty=false&filter_path=**._source&size=1000

then copy-paste the output to a JSON file and:

jq -c '.hits.hits[]' your-results.json | \
    elasticdump --input=$ --output="http://localhost:9200/your-new-index" --limit=1000

P.S. if you don't have jq, use sed instead:

sed -e 's/^{"hits":{"hits":\[//' -e 's/\]}}$//' -e 's/},{"_source":{/}\n{"_source":{/g'

Upvotes: 0

nicolas-f
nicolas-f

Reputation: 579

Using LogStash you can store elasticsearch query result into a new or existing index:

You have to combine elasticsearch input

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

And elastic search output

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html

Upvotes: 1

kub1x
kub1x

Reputation: 3552

Based on this post on ES forum, it is not possible to store query result directly back to ES as a new index. You have to fetch the data to client and push them into the new index yourself.

Upvotes: 2

Related Questions