CODEWITHSUNDEEP
elasticsearchfluentdamazon-opensearchtd-agent
S Andrew
S Andrew

Reputation: 7268

Index not found exception while sending data to Opensearch AWS using td agent

I have set up Opensearch in AWS. I have installed td-agent in Ubuntu 18.04. Below is my td-agent.conf file:

<source>
  @type tail
  path /home/rocket/PycharmProjects/EFK/log.json
  pos_file /home/rocket/PycharmProjects/EFK/log.json.pos
  format json
  time_format %Y-%m-%d %H:%M:%S
  tag log
</source>

<match *log*>
  @type opensearch
  host search-tanz-domain-2vbjmk2d4.us-west-2.es.amazonaws.com/
  port 9200
  scheme https
  ssl_verify false
  user admin
  password lah_001
  index_name test
</match>

When running the td-agent I am getting below error:

2023-01-26 15:41:44 +0000 [warn]: #0 Could not communicate to OpenSearch, resetting connection and trying again. [404] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [:9200]","index":":9200","resource.id":":9200","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [:9200]","index":":9200","resource.id":":9200","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}

So it's saying index not found which is a bit strange because as per my understanding when you send data to Opensearch or Elasticsearch then you need to create index pattern manually by using Kibana. I have never faced this error in Elasticsearch and I am only facing this issue in Opensearch while both of them looks to be same.

Edit

I have created the index using API:

enter image description here

I listed all the index and I can see test:

enter image description here

Now I again started uploading the data using td-agent but still getting the same error as above.

Upvotes: -1

Views: 1544

Answers (1)

Brian
Brian

Reputation: 1203

I haven't used td-agent before but based on the configuration file you provided it seems like it is trying to reach index test.

In opensearch, when you create a domain, it doesn't contain any indexes.

when you send data to opensearch or elasticsearch then you need to create index pattern manually by using kibana

I don't think this is necessarily true. You can create an index without using kibana and you can also create an index without sending any data. In fact, I think it is a better practice to create the index first and send the data later.

I think if you create the index test first it should work for you.

In Java:

OpenSearchClient client = new OpenSearchClient(
        new AwsSdk2Transport(
                httpClient,
                host,
                region,
                AwsSdk2TransportOptions.builder().build()));

// create the index
String index = "test";

CreateIndexRequest createIndexRequest = new CreateIndexRequest.Builder().index(index).build();
client.indices().create(createIndexRequest);

See: Sample code for Amazon OpenSearch Service

Upvotes: 1

Related Questions