CODEWITHSUNDEEP
elasticsearchlogstash
FrustratedWithFormsDesigner
FrustratedWithFormsDesigner

Reputation: 27486

Re-processing data for Elasticsearch with a new pipeline

I have an ELK-stack server that is being used to analyse Apache web log data. We're loading ALL of the logs, going back several years. The purpose is to look at some application-specific trends over this time period.

The data-processing pipeline is still being tweaked, as this is the first time anyone has looked in detail into this data and some people are still trying to decide how they want the data to be processed.

Some changes were suggested and while they're easy enough to do in the logstash pipeline for new, incoming data, I'm not sure how to apply these changes to the data that's already in elastic. It took several days to load the current data set, and quite a bit more data has been added so re-processing everything through logstash, with the modified pipeline will probably take several days longer.

What's the best way to apply these changes to data that has already been ingested into elastic? In the early stages of testing this set-up, I would just remove the index and rebuild from scratch, but that was done with very limited data sets and with the amount of data in use here, I'm not sure that's feasible. Is there a better way?

Upvotes: 1

Views: 1235

Answers (1)

ben5556
ben5556

Reputation: 3018

Setup an ingest pipeline and use reindex API to move data from current index to new index (with the pipeline configured for destination index)

Ingest Node

Upvotes: 3

Related Questions