CODEWITHSUNDEEP
elasticsearchlogstash
bilpor
bilpor

Reputation: 3889

logstash keeps placing just 1 entry in my index

I have the following logstash conf file:

input {
  jdbc {
    jdbc_driver_library => "C:\Program Files\Microsoft JDBC DRIVER 6.2 for SQL Server\sqljdbc_6.2\enu\mssql-jdbc-6.2.1.jre8"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://TST-DBS-20;user=Elasticsearch;password=elastic123;"
    jdbc_user => "Elasticsearch"
    statement => "SELECT NewsID, HeadLine, BodyText, DateSubmitted, Approved, NULLIF(UserName,'') as UserName, NULLIF(Type,'') as NewsType, NULLIF(Caption,'') as caption, NULLIF(Author,'') as Author, NULLIF(Contact,'') as Contact, NULLIF(StaffID,'') as StaffID, SocialClubRegionID, DateCreated, CreatedBy, LastModifiedDate, ModifiedBy
FROM [News].[dbo].[News]"
  }
}
filter {
}
output {
  elasticsearch {
    hosts => ["tst-sch-20:9200"]
    index => "newsindex"
    document_id => "%{id}"
    user => "elastic"
    password => elastic123
  }
  stdout { codec => json }
}

and I've created the following index:

put newsindex
{
  "settings" : {
    "number_of_shards":3,
    "number_of_replicas":2
  },
  "mappings" : {
    "news": {
      "properties": {
        "NewsId": {
          "type": "integer"
        },
        "newstype": {
          "type": "text"
        },
        "bodytext": {
          "type": "text"
        }
      }
    }
  }
}

After running the above script, there's no entry in the logstash log files to suggest anything went wrong. If I run the SQL command directly in SQL,then strangely enough, the single entry in the index is the last entry of my select statement, so it's almost as if the script is inserting then overwriting such that I end up with a single record.

Upvotes: 0

Views: 44

Answers (1)

Alcanzar
Alcanzar

Reputation: 17155

If you look at the _id field of the record loaded into Elasticsearch, you'll see it is %{id} because your query does not have an id field. You'll want to change to document_id => "%{newsid}" or whatever makes sense based on your query.

Upvotes: 1

Related Questions