CODEWITHSUNDEEP
mongodbelasticsearchlogstash
B2A3R9C9A
B2A3R9C9A

Reputation: 39

Sync data between Mongodb and ElasticSearch using Logstash

I am trying to sync my MongoDB database with elasticsearch and use the data on Kibana. I am using the plugins logstash-input-mongoDb and logstash-output-elasticsearch.

The issue is in the configuration file in Logstash to connect to MongoDB. Once I setup both plugins I created a config file to mention the input source details (MongoDB) and output (ElasticSearch). I have written it here:

input {
       uri => 'mongodb://localhost:27017/t_data/data'
       placeholder_db_dir => '/opt/logstash-mongodb/'
       placeholder_db_name => 'logstash_sqlite.db'
       collection => 't_data'
       batch_size => 5000
}

filter {
}

output {
        stdout {
                codec => rubydebug
       }
       elasticsearch {
               action => "index"
               index => "mongo_t_data"
               hosts => ["localhost:9200"]
       }
}

I then run this command from the bin directory in logstash where the config file is saved as mongodata.conf:

 \logstash-7.10.1\bin>logstash -f mongodata.conf

I obtain the following error:

 [2021-01-17T00:11:22,730][ERROR][logstash.agent           ] Failed to execute action 
 {:action=>LogStash::PipelineAction::Create/pipeline_id:main, 
 :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\" at 
 line 2, column 13 (byte 22) after input {\r\n        uri ", :backtrace=>["C:/PESU/logstash-7.10.1- 
 windows-x86_64/logstash-7.10.1/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", 
 "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", 
 "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "C:/PESU/logstash-7.10.1- 
 windows-x86_64/logstash-7.10.1/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", 
 "C:/PESU/logstash-7.10.1-windows-x86_64/logstash-7.10.1/logstash- 
 core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "C:/PESU/logstash-7.10.1-windows- 
 x86_64/logstash-7.10.1/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
 [2021-01-17T00:11:23,310][INFO ][logstash.agent           ] Successfully started Logstash API 
 endpoint {:port=>9600}
 [2021-01-17T00:11:28,026][INFO ][logstash.runner          ] Logstash shut down.
 [2021-01-17T00:11:28,058][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: 
 Logstash stopped processing because of an error: (SystemExit) exit

I have attached the screenshot here: enter image description here

My goal is to get logstash connected to Mongodata and elasticsearch succesfully to be able to sync data and add documents when required. I suspect it is an issue with me using Notepad to create the .conf file.

Upvotes: 0

Views: 1406

Answers (2)

R2D2
R2D2

Reputation: 10717

MongoDB since version 3.6 has feature called changesteams that can be used for this task, check here : https://medium.com/@montumodi/realtime-sync-from-mongodb-to-elasticsearch-using-change-streams-2559ad312011

Upvotes: 2

leandrojmp
leandrojmp

Reputation: 7463

Your input config is wrong. It is missing the type of input plugin.

It should be something like this:

input {
    mongodb {
        uri => 'mongodb://localhost:27017/t_data/data'
        placeholder_db_dir => '/opt/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 't_data'
        batch_size => 5000
    }
}

Try to change and see if it works.

Upvotes: 0

Related Questions