CODEWITHSUNDEEP
splunkmultilinefluentd
Achu
Achu

Reputation: 9

Handling multiline formats in splunk

I am not sure of how to set the BREAK_ONLY_BEFORE I have try setting

BREAK_ONLY_BEFORE: date

but it throws error

My logs are :

[2022-04-05 11:18:23,839] WARN Error while loading: connectors-versions.properties (com.amadeus.scp.kafka.connect.utils.Version)
java.lang.NullPointerException
    at java.util.Properties$LineReader.readLine(Properties.java:434)
    at java.util.Properties.load0(Properties.java:353)
    at java.util.Properties.load(Properties.java:341)
    at com.amadeus.scp.kafka.connect.utils.Version.<clinit>(Version.java:47)
    at com.amadeus.scp.kafka.connect.connectors.kafka.source.router.K2KRouterSourceConnector.version(K2KRouterSourceConnector.java:62)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:380)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:385)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:355)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:328)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:261)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:253)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:222)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:199)
    at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:60)
    at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)

As of now the logs are send to splunk as different events. Please let me know if anyone has used this option before.

Upvotes: 0

Views: 910

Answers (1)

RichG
RichG

Reputation: 9926

The better method is to set LINE_BREAKER to a regular expression that defines the start of each event. These props.conf settings should do it.

[mysourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\[\d{4}-\d\d
TIME_PREFIX = \[
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 23

If you really want to use BREAK_ONLY_BEFORE then try these settings:

[mysourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ([\r\n]+)\[\d{4}-\d\d
TIME_PREFIX = \[
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 23

Upvotes: 1

Related Questions