CODEWITHSUNDEEP
elasticsearchlucenerequestkibana
Julen Cestero
Julen Cestero

Reputation: 95

Create index-patterns from console with Kibana 6.0 or 7+ (v7.0.1)

I recently upgraded my ElasticStack instance from 5.5 to 6.0, and it seems that some of the breaking changes of this version have harmed my pipeline. I had a script that, depending on the indices inside ElasticSearch, created index-patterns automatically for some groups of similar indices. The problem is that with the new mapping changes of the 6.0 version, I cannot add any new index-pattern from the console. This was the request I used and worked fine in 5.5:

curl -XPOST "http://localhost:9200/.kibana/index-pattern" -H 'Content-  Type: application/json' -d'
{
  "title" : "index_name",
  "timeFieldName" : "execution_time"
}'

This is the response I get now, in 6.0, from ElasticSearch:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [index-pattern, doc]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [index-pattern, doc]"
  },
  "status": 400
}

How could I add index-patterns from the console avoiding this multiple mapping issue?

Upvotes: 8

Views: 11850

Answers (5)

Tarasovych
Tarasovych

Reputation: 2398

For Kibana 7.7.0 with Open Distro security plugin (amazon/opendistro-for-elasticsearch-kibana:1.8.0 Docker image to be precise), this worked for me:

curl -X POST \
-u USERNAME:PASSWORD \
KIBANA_HOST/api/saved_objects/index-pattern \
-H "kbn-version: 7.7.0" \
-H "kbn-xsrf: true" \
-H "content-type: application/json; charset=utf-8" \
-d '{"attributes":{"title":"INDEX-PATTERN*","timeFieldName":"@timestamp","fields":"[]"}}'

Please note, that kbn-xsrf header is required, but it seems like it's useless as from security point of view.

Output was like:

{"type":"index-pattern","id":"UUID","attributes":{"title":"INDEX-PATTERN*","timeFieldName":"@timestamp","fields":"[]"},"references":[],"migrationVersion":{"index-pattern":"7.6.0"},"updated_at":"TIMESTAMP","version":"VERSION"}

I can't tell why migrationVersion.index-pattern is "7.6.0".

For other Kibana versions you should be able to:

  1. Open Kibana UI in browser
  2. Open Developers console, navigate to Network tab
  3. Create index pattern using UI
  4. Open POST request in the Developers console, take a look on URL and headers, than rewrite it to cURL

Upvotes: 1

NOZUONOHIGH
NOZUONOHIGH

Reputation: 2006

Create index-pattern in bulk with timestamp：

cat index_svc.txt
my-index1
my-index2
my-index3
my-index4
my-index5
my-index6

cat index_svc.txt | while read index; do
    echo -ne "create index-pattern ${index} \t"
    curl -XPOST "http://10.0.1.44:9200/.kibana/doc/index-pattern:${index}" -H 'Content-Type: application/json' -d "{\"type\":\"index-pattern\",\"index-pattern\":{\"title\":\"${index}2020*\",\"timeFieldName\":\"@timestamp\"}}"
    echo 
done

Upvotes: 0

AKS
AKS

Reputation: 17336

If you are Kibana 7.0.1 / 7+ then you can refer saved_objects API ex: Refer: https://www.elastic.co/guide/en/kibana/master/saved-objects-api.html (Look for Get, Create, Delete etc).

In this case, we'll use: https://www.elastic.co/guide/en/kibana/master/saved-objects-api-create.html

$ curl -X POST -u $user:$pass -H "Content-Type: application/json" -H "kbn-xsrf:true"  "${KIBANA_URL}/api/saved_objects/index-pattern/dummy_index_pattern" -d '{ "attributes": { "title":"index_name*", "timeFieldName":"sprint_start_date"}}'  -w "\n" | jq

and

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   327  100   250  100    77    543    167 --:--:-- --:--:-- --:--:--   543
{
  "type": "index-pattern",
  "id": "dummy_index_pattern",
  "attributes": {
    "title": "index_name*",
    "timeFieldName": "sprint_start_date"
  },
  "references": [],
  "migrationVersion": {
    "index-pattern": "6.5.0"
  },
  "updated_at": "2020-02-25T22:56:44.531Z",
  "version": "Wzg5NCwxNV0="
}

Where $KIBANA_URL was set to: http://my-elk-stack.devops.local:5601

If you don't have jq installed, remove | jq from the command (as listed above).

PS: When KIBANA's GUI is used to create an index-pattern, Kibana stores its i.e. index ID as an alpha-numeric value (ex: laskl32ukdflsdjflskadf-sdf-sdfsaldkjfhsdf-dsfasdf) which is hard to use/find/type when doing GET operation to find info about an existing index-pattern using the following curl command.

If you passed index pattern name (like we did above), then in Kibana/Elasticsearch, it'll story the Index-Pattern's ID by the name you gave to the REST call (ex: .../api/saved_objects/index-pattern/dummy_index_pattern")

here: dummy_index_pattern will become the ID (only visible if you hover over your mouse on the index-pattern name in Kibana GUI) and

it'll have it's index name as: index_name* (i.e. what's listed in GUI when you click on Kibana Home > Gear icon > Index Patterns and see the index patterns listed on the right side.

NOTE: The timeFieldName is very important. This is the field, which is used for looking for time-series events (i.e. especially TSVB Time Series Visual Builder Visualization type). By default, it uses @timestamp field, but if you recreate your index (instead of sending delta information to your target Elasticsearch index from a data source (ex: JIRA)) every time and send all data in one shot from scratch from a data source, then @timestamp won't help with Visualization's Time-Spanning/Window feature (where you change time from last 1 week to last 1 hour or last 6 months); in that case, you can set a different field i.e. sprint_start_date like I used (and now in Kibana Discover data page, if you select this index-pattern, it'll use sprint_start_date (type: date) field, for events.

To GET index pattern info about the newly created index-pattern, you can refer: https://www.elastic.co/guide/en/kibana/master/saved-objects-api-get.html --OR run the following where (the last value in the URL path is the ID value of the index pattern we created earlier:

curl -X GET "${KIBANA_URL}/api/saved_objects/index-pattern/dummy_index_pattern" | jq

or

otherwise (if you want to perform a GET on an index pattern which is created via Kibana's GUI/webpage under Page Index Pattern > Create Index Pattern, you'd have to enter something like this:

curl -X GET "${KIBANA_URL}/api/saved_objects/index-pattern/jqlaskl32ukdflsdjflskadf-sdf-sdfsaldkjfhsdf-dsfasdf" | jq

Upvotes: 1

Sanoussy Diallo
Sanoussy Diallo

Reputation: 106

The URL has been changed in version 6.0.0, here is the new URL:

http://localhost:9200/.kibana/doc/doc:index-pattern:my-index-pattern-name

This CURL should work for you:

curl -XPOST "http://localhost:9200/.kibana/doc/index-pattern:my-index-pattern-name" -H 'Content-Type: application/json' -d'
{
  "type" : "index-pattern",
  "index-pattern" : {
    "title": "my-index-pattern-name*",
    "timeFieldName": "execution_time"
  }
}'

Upvotes: 9

emhlbmc
emhlbmc

Reputation: 258

Indices created in Elasticsearch 6.0.0 or later may only contain a single mapping type.

Indices created in 5.x with multiple mapping types will continue to function as before in Elasticsearch 6.x.

Mapping types will be completely removed in Elasticsearch 7.0.0.

Maybe you are creating a index with more than one doc_types in ES 6.0.0. https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

Upvotes: 0

Related Questions