CODEWITHSUNDEEP
encodinglogstashrsyslog
Paradise
Paradise

Reputation: 1468

rsyslog sending badly encoded (corrupted?) data via tcp (receiving using logstash)

My rsyslog logs locally correctly, however I wanted to also receive the logs remotely, so I added the rule:

*.* @@myIP:5141

to the end of my rsyslog.conf

To receive the output, I'm running logstash with the configuration

input { tcp { port => 5141 } }
output { stdout {} }

Logstash expects UTF-8 encoding, however I get the error

Received an event that has a different character encoding than you configured

The messages themselves seem to be garbled, or a mix of encodings, for example:

\u0016\u0003\u0002\u0000V\u0001\u0000\u0000R\u0003\u0002S\xB1R\xAB5K\xF6\\\xB9\xB2\xB4\xB1\xAE0\t\u007F\xDF`5\xF6\u0015\xC8)H\xD7H\xCF+&\xD5T5\u0000\u0000$\u00003\u0000E\u00009\u0000\x88\u0000\u0016\u00002\u0000D\u00008\u0000\x87\u0000\u0013\u0000f\u0000/\u0000A\u00005\u0000\x84\u0000

Note some entries are \u00, while others are \x. There are even multiple backslashes.

I was wondering if I messed up the settings somehow, or if there is something between me and the server which is messing up the messages?

I have also tried using the syslog logstash input, which gives the same result

Another example:

\u0016\u0003\u0002\u0000V\u0001\u0000\u0000R\u0003\u0002S\xB1RiZ^\xC3\xD9\u001Cj\a\xD4\xE0\xECr\x8E\xAC\xF5\u001A\xB9+\u07B9\xE5\xF9\xA3''z\u0018}9\u0000\u0000$\u00003\u0000E\u00009\u0000\x88\u0000\u0016\u00002\u0000D\u00008\u0000\x87\u0000\u0013\u0000f\u0000/\u0000A\u00005\u0000\x84\u0000

EDIT: I found the source of my problem, and it was encryption related. Unfortunately I can't disclose what I did to fix it, suffice to say John Petrone's answer below is good start for similar problems that future readers may experience

Upvotes: 4

Views: 10301

Answers (3)

mlibre
mlibre

Reputation: 2540

based on @docwhat answer.

nano logstash/pipeline/logstash.conf
# Or
nano /path/to/logstash.conf
input {
 beats {
      port => 5000
      ssl => false
 }
  #tcp {
  #   port => 5000
  #}
}

Upvotes: 0

docwhat
docwhat

Reputation: 11704

So that magic string you're getting back that looks like broken encoding is actually the SSL Handshake request.

I suspect what you've done is (like I just did) misconfigured the tcp input in logstash. Specifically, I forgot to add the ssl_enable => true. So it was listening for normal TCP and got SSL Handshake and dutifully recorded it as garbage.

Upvotes: 7

John Petrone
John Petrone

Reputation: 27497

The problem is that a syslog source that you are ingesting is sending data in non UTF-8 format which is causing problems with Logstash, as that is what it is expecting. You've basically got 3 courses of action:

  1. Have Rsyslog correct this for you: Use the Rsyslog mmutf8fix module to fix invalid UTF-8 sequences. http://www.rsyslog.com/doc/mmutf8fix.html

  2. Change Logstash to use a more appropriate charset: You can change the default charset for the plain codec: http://logstash.net/docs/1.4.2/codecs/plain . You will need to experiment a bit, I'd check here for a starting point. https://logstash.jira.com/browse/LOGSTASH-1047

  3. Change your source to output UTF-8: Not knowing the sources being collected by Rsyslog I can't comment on what it would take to make this change.

I'd start with option 1 and if that does not work move to option 2.

Upvotes: 1

Related Questions