CODEWITHSUNDEEP
mysqllogstashelastic-stack
Karthik
Karthik

Reputation: 343

Logstash mysql database data not loading to elasticsearch index

I have been working on this all evening and it is driving me crazy. It is supposed to be very simple but it is not working. This works with Oracle but not with MySQL and I created similar db.config that is fed to logstash using -f option.

input {

 jdbc {
       jdbc_driver_library => "/opt/elk/logstash-5.6.0/lib/mysql-connector-java-5.1.45-bin.jar"
       jdbc_driver_class => "Java::com.mysql.jdbc.Driver"
       jdbc_connection_string => "jdbc:mysql://serverName:3306/dbName?verifyServerCertificate=false&useSSL=true"
       jdbc_user => "userName"
       jdbc_password => "PasswordValue"

    statement => "select user_id, visitor_returning, config_os, visitor_days_since_last from visiting_table where user_id is not null"

    #optional extras I use 
      type => "visit"
      tags => ["awesome", "import"]
 }

}

output{ stdout {codec => json_lines}

if [type] == "visit"{   
        elasticsearch{
            hosts => "127.0.0.1"
            index => "visitDb"
            document_type => "visit_results"
        }
    }
stdout{}

}

Once I run the logstash, it is not loading the data into Elastic search index. I cannot even see the index named visitDb when I do the below.

curl 'localhost:9200/_cat/indices?v'

health status index          uuid                   pri rep docs.count docs.deleted store.size pri.store.size

yellow open .kibana Fg6P7XuHSTaonbKEbLcz5A 1 1 21 3 56.5kb 56.5kb yellow open orderstotdb obxZ38prTFCG0W-BFTIhgw 5 1 60 0 245.4kb 245.4kb y

I am unable to figure out what is going on with MySQL. I can see the console log retrieving the data in json format and writing to console, but the index does not reflect in ElasticSearch nor shows up in Kibana to create index pattern.

Can someone please help ??

Upvotes: 0

Views: 662

Answers (1)

Karthik
Karthik

Reputation: 343

Answering my own question in case any one else is having the same issue. Elastic Search rejects index names with upper case, since I had name visitDb with D capital, it was rejecting index name creation, go figure :) Hours lost in debugging different options.

Upvotes: 0

Related Questions