Reputation: 367
I am new to ELK stack. Stuck with data ingestion in elasticsearch using logstash.
I am mentioning the steps I followed:- Installed ELK Stack successfully. Then installed plugin logstash-input-mongodb. After that configured logstash with below file:-
input {
mongodb {
uri => 'mongodb://localhost:27017/dbName'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'notifications'
batch_size => 1
}
}
output {
elasticsearch {
action => "index"
index => "notifications_data"
hosts => ["localhost:9200"]
}
stdout { codec => json }
}
Saved the above file as mongo-connector.conf
then run this using
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/mongo-connector.conf
After this, the logs on the terminal was:-
D, [2020-11-07T14:01:45.739178 #29918] DEBUG -- : MONGODB | localhost:27017 req:480 conn:1:1 sconn:78 | dbName.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"dbName", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x4508feda2dce4ec6...>}}
D, [2020-11-07T14:01:45.741919 #29918] DEBUG -- : MONGODB | localhost:27017 req:480 | dbName.listCollections | SUCCEEDED | 0.002s
D, [2020-11-07T14:01:50.756430 #29918] DEBUG -- : MONGODB | localhost:27017 req:481 conn:1:1 sconn:78 | dbName.find | STARTED | {"find"=>"notifications", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5fa012440d0e947dd8dfd2f9')}}, "limit"=>1, "$db"=>"dbName", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x4508feda2dce4ec6...>}}
D, [2020-11-07T14:01:50.758080 #29918] DEBUG -- : MONGODB | localhost:27017 req:481 | dbName.find | SUCCEEDED | 0.001s
D, [2020-11-07T14:01:50.780259 #29918] DEBUG -- : MONGODB | localhost:27017 req:482 conn:1:1 sconn:78 | dbName.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"dbName", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x4508feda2dce4ec6...>}}
D, [2020-11-07T14:01:50.782687 #29918] DEBUG -- : MONGODB | localhost:27017 req:482 | dbName.listCollections | SUCCEEDED | 0.002s
D, [2020-11-07T14:01:53.986862 #29918] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'standalone' to 'standalone' [awaited].
D, [2020-11-07T14:01:53.987784 #29918] DEBUG -- : MONGODB | There was a change in the members of the 'Single' topology.
D, [2020-11-07T14:01:54.311966 #29918] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'standalone' to 'standalone'.
D, [2020-11-07T14:01:54.312747 #29918] DEBUG -- : MONGODB | There was a change in the members of the 'Single' topology.
D, [2020-11-07T14:01:55.799418 #29918] DEBUG -- : MONGODB | localhost:27017 req:483 conn:1:1 sconn:78 | dbName.find | STARTED | {"find"=>"notifications", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5fa012440d0e947dd8dfd2f9')}}, "limit"=>1, "$db"=>"dbName", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x4508feda2dce4ec6...>}}
Below is the logstash logs file:-
[2020-11-07T16:32:33,678][WARN ][logstash.inputs.mongodb ][main][6a52e3ca90ba4ebc63108d49c11fcede25b196c679f313b40b02a8e17606c977] MongoDB Input threw an exception, restarting {:exception=>#<Sequel::DatabaseError: Java::JavaSql::SQLException: attempt to write a readonly database>}
Index gets created on elasticsearch but docs don't get inserted there.
Upvotes: 0
Views: 687