CODEWITHSUNDEEP
splunk
Peter Rubi
Peter Rubi

Reputation: 119

Splunk forwarder gets blocked if one output destination is down

I have a Splunk forwarder which sends events to two third-party systems(through TCP) and also index them into a splunk indexer.

The problem I'm facing on is that if any of the two third-pary systems goes down. Splunk stops indexing events and neither sends them to the other system.

The output.conf I have is:

[tcpout]
defaultGroup = default-system1, default-system2
indexAndForward = 0

[tcpout:default-system1]
server = <IP>:<PORT>

[tcpout-server://<IP>:<PORT>]

[tcpout:default-system2]
server = <IP>:<PORT>
sendCookedData = false

Is there any way to avoid such a dependency? If one of the destinatios servers is down, it doesn't affect the other sending. I've been looking at the documentation and there are some options that could be use.

heartbeatFrequency in combination with sendCookedData.

heartbeatFrequency = <integer>
* How often (in seconds) to send a heartbeat packet to the receiving server.
* This setting is a mechanism for the forwarder to know that the receiver
  (indexer) is alive. If the indexer does not send a return packet to the
  forwarder, the forwarder declares the receiver unreachable and does not
  forward data to it.
* The forwarder only sends heartbeats if the 'sendCookedData' setting
  is set to "true".
* Default: 30


sendCookedData = <boolean>
* Whether or not to send processed or unprocessed data to the receiving server.
* If set to "true", events are cooked (have been processed by Splunk software).
* If set to "false", events are raw and untouched prior to sending.
* Set to "false" if you are sending events to a third-party system.
* Default: true

But I'm not sure if it's the most correct approach, based on the description of sendCookedData, "Set to "false" if you are sending events to a third-party system."

Upvotes: 1

Views: 2876

Answers (1)

RichG
RichG

Reputation: 9976

Don't send cooked data to third-party systems. That way lies sadness.

Unfortunately, the behavior you describe is normal for Splunk. From all outward appearances, there is a single output queue and all destinations are fed from that one queue. To avoid data loss, all sending stops when one destination is unavailable.

Upvotes: 1

Related Questions