CODEWITHSUNDEEP
postgresqlelasticsearchlogstash
Stephan
Stephan

Reputation: 3799

logstash out of memory reading postgres large table

Im trying to index a Large database table with more then 10.000.000 AND logstash is running out of memory.. :(

The Error:

logstash_1       | Error: Your application used more memory than the safety cap of 1G.
logstash_1       | Specify -J-Xmx####m to increase it (#### = cap size in MB).
logstash_1       | Specify -w for full OutOfMemoryError stack trace

My logstash configuration:

input {
    jdbc {
        # Postgres jdbc connection string to our database, mydb
        jdbc_connection_string => "jdbc:postgresql://database:5432/predictiveparking"
        # The user we wish to execute our statement as
        jdbc_user => "predictiveparking"
        jdbc_password => "insecure"
        # The path to our downloaded jdbc driver
        jdbc_driver_library => "/app/postgresql-9.4.1212.jar"
        # The name of the driver class for Postgresql
        jdbc_driver_class => "org.postgresql.Driver"
        # our query
        statement => "SELECT * from scans_scan limit 10"
    }
}


#output {
#    stdout { codec => json_lines }
#}

output {
    elasticsearch {
    index => "scans"
    sniffing => false
    document_type => "scan"
    document_id => "id"
    hosts => ["elasticsearch"]
    }
}

Upvotes: 0

Views: 942

Answers (1)

Stephan
Stephan

Reputation: 3799

Just enabling paging..

added: 

jdbc_paging_enabled => true

Now the data form database get's cut into pieces and we do not run out of memory. Make sure the sql query is ORDERED!

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#plugins-inputs-jdbc-jdbc_paging_enabled

Upvotes: 1

Related Questions