CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2logstash
user3362408
user3362408

Reputation: 41

How to use Logstash Forwarder to Send logs to Different Nodes on AWS

We want to use Logstash forwarder (previously known as Lumberjack) to push logs to a central Logstash indexer. The indexer, in turn, would feed the log entries into Elastic Search, located on the same server as the indexer. We have successfully implemented the setup on virtual machines running on a single computer, and now want to set it up on our AWS instances. We want the forwarder to live on one EC2 instance while the indexer and Elastic Search live on another EC2 instance. Our config files look like this:

Logstash Forwarder config:

{
"network": {
    "servers": [
        "public.facing.url.com:5555"
    ],
    "ssl ca": "logstash-forwarder.crt",
    "timeout": 15
},
"files": [
    {
        "paths": [
            "messages",
            "/var/log/tomcat7/*.log"
        ],
        "fields": {
            "type": "syslog"
        }
    },
]
}

Logstash (as indexer) config:

input{
lumberjack{
    port=>5555
    ssl_certificate=>"logstash-forwarder.crt"
    ssl_key=>"logstash-forwarder.key"
    type=>"linkoslogs"
}stdin{
    type=>example
}
}output{
    elasticsearch{
        host=>"127.0.0.1"
    }
}

Logstash (indexer) starts up fine, with no exceptions. It indicates that it is ready and waiting for Lumberjack on port 5555. Logstash forwarder, however, fails on startup:

2014/02/27 21:29:53.797935 Connecting to 1.2.3.4:5555 (public.facing.url.com)
2014/02/27 21:30:08.798198 Failure connecting to 1.2.3.4: dial tcp 1.2.3.4:5555: i/o timeout
2014/02/27 21:30:09.799179 Connecting to 1.2.3.4:5555 (public.facing.url.com)
... (repeats itself indefinitely)

Do I have to explicitly instruct AWS to allow communication over port 5555 when it's another AWS instance trying to initiate communication? I can't think of what else would be preventing the forwarder from bridging the gap to the indexer.

Upvotes: 4

Views: 3073

Answers (1)

icalvete
icalvete

Reputation: 1159

Open 5555 port for the security group where Logstash Forwarder is run on security group where Logstash Indexer is run (Even if same security group for both !!!)

Use Group ID value. For instance sg-18e1f37a

Upvotes: 2

Related Questions