CODEWITHSUNDEEP
elasticsearchlogstash
Ram Bhatt
Ram Bhatt

Reputation: 91

Logstash is skipping records while inserting records in elastic search

I am new to elastic search. I am using to Logstash to push data from my PostgreSQL Database to elastic index. I usually set the jdbc_page_size => 100000 in the config file for faster ingestion. However, data is not fully pushed even if logstash logs say all data has been pushed. So, I set jdbc_page_size => 25000, which solves my problem

I am facing this problem particularly with PostgesSQL(not with MySQL or MS SQL Server). If anyone has any insight, please clarify why this is happening.

EDIT : Config File as requested:

input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://ip:5432/dbname"
jdbc_user => "postgres"
jdbc_password => "postgres"
jdbc_driver_library => "/postgresql.jar"
jdbc_driver_class => "org.postgresql.Driver"
jdbc_paging_enabled => true
jdbc_page_size => 25000
statement => "select * from source_table"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "sample"
document_type => "docs"
document_id => "%{id}"
}
}

Upvotes: 0

Views: 809

Answers (1)

Shashank Gutha
Shashank Gutha

Reputation: 145

PostgreSQL does not give records in same order so kindly add order by clause in query, it will solve your issue. you can try below configuration, it's working.

input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://ip:5432/dbname"
jdbc_user => "postgres"
jdbc_password => "postgres"
jdbc_driver_library => "/postgresql.jar"
jdbc_driver_class => "org.postgresql.Driver"
jdbc_paging_enabled => true
jdbc_page_size => 25000
statement => "select * from source_table order by id desc"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "sample"
document_type => "docs"
document_id => "%{id}"
}
}

Upvotes: 1

Related Questions