CODEWITHSUNDEEP
.netelasticsearchserilog
Akhil Jose
Akhil Jose

Reputation: 101

Pipeline specified in the settings of serilog-elasticsearch is not executed

I am trying to write logs from my .Net application to Elasticsearch using the serilog-elasticsearch sink . I was able to create indexes dynamically and write logs.

I was using the app.settings method for serilog-elastic. Following is my app-settings, in the .Net project from which I create logs.

 <add key="serilog:using" value="Serilog.Sinks.Elasticsearch" />
<add key="serilog:write-to:Elasticsearch.nodeUris" value="http://localhost:9200" />
<add key="serilog:write-to:Elasticsearch.indexFormat" value="app-index-{0:yyyy.MM.dd}" />
<!--<add key="serilog:write-to:Elasticsearch.templateName" value="apptemplate" />--> 
<add key="serilog:write-to:Elasticsearch.typeName" value="appLogEvent" />
<add key="serilog:write-to:Elasticsearch.pipelineName" value="geoip" />
<add key="serilog:write-to:Elasticsearch.batchPostingLimit" value="50" />
<add key="serilog:write-to:Elasticsearch.emitEventFailure" value="WriteToSelfLog" />
<add key="serilog:write-to:Elasticsearch.period" value="2" />
<add key="serilog:write-to:Elasticsearch.inlineFields" value="true" />
<add key="serilog:write-to:Elasticsearch.minimumLogEventLevel" value="Debug" />
<!--<add key="serilog:write-to:Elasticsearch.bufferBaseFilename" value="C:\Logs\elastic-buffer.log" />--> 
<add key="serilog:write-to:Elasticsearch.bufferFileSizeLimitBytes" value="5242880" />
<add key="serilog:write-to:Elasticsearch.bufferLogShippingInterval" value="5000" />
 <!--<add key="serilog:write-to:Elasticsearch.connectionGlobalHeaders" value="Authorization=Bearer SOME-TOKEN;OtherHeader=OTHER-HEADER-VALUE" />--> 
<add key="logPath" value="C:\Logs\App" />
<add key="logLevel" value="Error" />
<!-- Log Rolling: File size limit for roll over (code defaults to 50MB if not specified here) -->
<add key="logRollFileSizeLimit" value="52428800" />
<!-- Log Rolling: Number of files to retain when rolled over (defaults to 10 if not specified here) -->
<add key="logRollFileRetainCount" value="10" />

I needed to apply a plugin geoip-processor on the indexed documents and identify the IP addresses in the log and convert them to geo-codes.

I created pipeline to execute processors and map the necessary data. I was able to test it using the dev-console tools. But when I applied the same pipeline on the documents by specifying in the app.settings, it does not work. I created different pipelines and tried, but with little success. Following is my pipeline:

{ "geoip" : { "description" : "Add geo ip info", "processors" : [ { "set" : { "field" : "IP1", "value" : "{{response.IP}}" } }, { "trim" : { "field" : "IP1" } }, { "geoip" : { "field" : "IP1", "target_field" : "geo_address", "ignore_missing" : true, "database_file" : "GeoLite2-Country.mmdb" } } ] } }

I checked the elastic logs, but nothing suspicious was found. The version of elasticsearch is: 6.6.0 .Net version is 4.5.2 Other version's of serilog related nuget packages from packages.config file are:

<package id="ElasticSearch.Net" version="5.5.0" targetFramework="net452" /> <package id="Serilog" version="2.5.0" targetFramework="net452" /> <package id="Serilog.Formatting.Compact" version="1.0.0" targetFramework="net452" /> <package id="Serilog.Settings.AppSettings" version="2.1.0" targetFramework="net452" /> <package id="Serilog.Sinks.ElasticSearch" version="5.4.0" targetFramework="net452" /> <package id="Serilog.Sinks.File" version="4.0.0" targetFramework="net452" /> <package id="Serilog.Sinks.PeriodicBatching" version="2.1.1" targetFramework="net452" /> <package id="Serilog.Sinks.RollingFile" version="3.3.0" targetFramework="net452" /> <package id="Serilog.Sinks.Seq" version="3.3.3" targetFramework="net452" />

Upvotes: 0

Views: 1237

Answers (1)

Akhil Jose
Akhil Jose

Reputation: 101

The issue was because of the version conflict of elasticsearch and Serilog.Sinks.Elasticsearch. It seems that the version needed to be updated to the latest. I was using Serilog.Sinks.Elasticsearch version=5.4.0 and updated to the latest (7.1.0). The pipeline is now executing as intended. The ElasticSearch version was 6.6.0.

Unfortunately the version compatibility was never specified anywhere in the github page

Upvotes: 1

Related Questions