CODEWITHSUNDEEP
loggingelasticsearch
guettli
guettli

Reputation: 27855

Storing structured logs in elasticsearch

We consider to store structered logs in elasticsearch. This is a high level question of someone new to elasticsearch.

I am unsure if we can delete the structured logs after indexing them in elasticsearch.

I see no reason why the logs should be kept. They could be retrieved from elasticsearch again.

As long as we have enough storage space for elasticsearch, why should we keep a copy of the logs outside of elasticsearch?

Upvotes: 4

Views: 10494

Answers (2)

guettli
guettli

Reputation: 27855

Quoting the docs:

The _source field is an automatically generated field that stores the actual JSON that was used as the indexed document. It is not indexed (searchable), just stored.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html

If you have backups of elasticsearch and if you don't disable the _source field, you can drop/delete/remove the original json data.

You are free to move old stuff from elasticsearch to a different storage after N years.

Answer: You don't need to store the data else where.

Upvotes: 2

ryan1234
ryan1234

Reputation: 7275

A quick note about structured logs in ES ... I would recommend looking at the ELK stack: http://www.elasticsearch.org/webinars/elk-stack-devops-environment/

It uses logstash at the center of shipping logs, which is a pretty tried and true technology.

To get up and running really fast, you can use a Docker image (https://registry.hub.docker.com/u/qnib/elk/) which has ES, logstash and Kibana installed for you. Saves a lot of headaches trying to figure out how to do it yourself.

As for keeping logs ... I would only keep a small window of logs (2 weeks?) on a machine. Use logstash forwarder as a client (https://github.com/elasticsearch/logstash-forwarder) and it will automatically rotate logs for you. You'd want the logs on the machine in case ES goes down.

Upvotes: 4

Related Questions