CODEWITHSUNDEEP
logstashlogstash-input-jdbc
Frank
Frank

Reputation: 3

Logstash sequential thread execution flow

I made two jdbc sections in my config, but I want always execute first actionA and than actionB, it's important for me. How can I do this?

Here is my config:

input {
    jdbc {
        type => "actionA"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        statement => "SELECT ID FROM my_table WHERE delete='Y'"
       }

    jdbc {
        type => "actionB"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        statement => "UPDATE my_table SET delete='T' WHERE delete='Y'"
       }
}
output {
    stdout { codec => rubydebug }
    if [type] == "actionA" {
        elasticsearch {
            action => "delete"
            index => "contacts"
            document_type => "contact"
            document_id => "%{id}"
            hosts => ["http://localhost:9200"]
        }   
    }
    if [type] == "actionB" { }


}

Thanks in advance

Jay

Upvotes: 0

Views: 396

Answers (2)

rkok
rkok

Reputation: 1064

There is a trick for it.

Put your actions in separate files, so each file looks like this:

# action_X.conf
input {
    jdbc {
        # ...
    }
}
output {
    # ...
}

Then execute logstash manually for each file:

logstash -f action_a.conf
logstash -f action_b.conf

To repeat periodically, you could put the above commands in a shell script and add that script to the crontab.

Upvotes: 0

Val
Val

Reputation: 217304

You can leverage the schedule parameter and make actionA always run before actionB, e.g. by making actionA run every even minutes (0, 2, 4, 6, ...) and actionB run every odd minutes (1, 3, 5, ...)

input {
    jdbc {
        type => "actionA"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        schedule => "*/2 * * * *" # run every even minutes
        statement => "SELECT ID FROM my_table WHERE delete='Y'"
       }

    jdbc {
        type => "actionB"
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@mycomp:1522/db1"
        ...
        schedule => "1-59/2 * * * *" # run every odd minutes
        statement => "UPDATE my_table SET delete='T' WHERE delete='Y'"
       }
}

Upvotes: 0

Related Questions