CODEWITHSUNDEEP
logstash
kuronue
kuronue

Reputation: 306

Logstash to output events in Elasticsearch bulk API data format

Is is possible to have Logstash to output events in Elasticsearch bulk API data format?

The idea is to do some heavy parsing on many machines (without direct connectivity to the ES node) and then feed the data manually into ES.

Thank for the help.

Upvotes: 1

Views: 2559

Answers (2)

Gerardo Rochín
Gerardo Rochín

Reputation: 309

Maybe if you need change the flush_size in Logstash with your value:

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-flush_size

Or send metadata in file using json codec and afterload directly on elasticsearch https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html

Upvotes: 1

Alain Collins
Alain Collins

Reputation: 16362

Logstash is a single-line type of system, and the bulk format is a multi-line format. Here are two ideas:

1) see if the file{} output message_format can contain a newline. This would allow you to output the meta data line and then the data line.

2) use logstash's clone{} to make a copy of each event. In the "original" event, use the file{} output with a message_format that looks like the first line of the bulk output (index, type, id). In the cloned copy, the default file{} output might work (or use the message_format with the exact format you need).

Upvotes: 0

Related Questions