CODEWITHSUNDEEP
indexingelasticsearchlogstash
Abhinav Choudhary
Abhinav Choudhary

Reputation: 41

Logstash not writing data from external file to elasticsearch

I have a sample text file named testfile.txt containing simple "Hi". I want this data to get indexed on ElasticSearch. I run the following command on logstash:

bin/logstash -f logstash-test.conf

Conf File content is below:

input{
    file
        {
            path=> "/home/abhinav/ELK/logstash/testfile.txt"
            type => "test"
            start_position => "beginning"
        }
     }
output{
     elasticsearch { host => localhost 
             index => "test_index"
               }
     stdout { codec => rubydebug } 
      }

The ElasticSearch Log shows the follwing error:

[2015-05-04 14:52:23,082][INFO ][cluster.service ] [Argo] added {[logstash-abhinav-HP-15-Notebook-PC-10919-4006][CPk1djqFRnO-j-DlUMJIzg][abhinav-HP-15-Notebook-PC][inet[/192.168.1.2:9301]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[logstash-abhinav-HP-15-Notebook-PC-10919-4006][CPk1djqFRnO-j-DlUMJIzg][abhinav-HP-15-Notebook-PC][inet[/192.168.1.2:9301]]{client=true, data=false}])

I tried following things:

Tried with simple std input(stdin) to ES n stdout . It worked.

Upvotes: 2

Views: 1492

Answers (2)

Abhinav Choudhary
Abhinav Choudhary

Reputation: 41

I have found solution to my problem and also found things we can check, if logstash & ES not working/communicating properly. :

  • Make sure ES & Logstash are properly installed
  • Versions installed are compatible with each other
  • Even while testing, try making different Logstash conf file for different test case, as suggested by Mr. Alcanzar in above comment.

You can also refer to below links , for help regarding this issue :

Cannot load index to elasticsearch from external file, using logstash

https://logstash.jira.com/browse/LOGSTASH-1800

Upvotes: 0

Alcanzar
Alcanzar

Reputation: 17155

If you are using the same file repeatedly to test with, you are running into the "sincedb" problem -- see How to force Logstash to reparse a file?

You need to add sincedb_path => "/dev/null" to your file input. Generally this is not needed in a production scenario, but it is when testing.

Upvotes: 2

Related Questions