How to logging in Amazon Web Service ( AWS )?

Question

I have a project built in Golang and deployed on a Docker instance in AWS.

Internally I create a log file where the program write several logs.

How can I access that log file?

Is there another correct way to logging?

Thanks

Answer 1

If you use ELK (Elasticsearch, Logstash, Kibana), I would suggest to use "logrus"

Get the library

go get github.com/sirupsen/logrus

Then in your project

package main

import (

    logrus "github.com/sirupsen/logrus"
)

var log = logrus.New()

func main() {

    conn, _ := net.Dial("tcp", "logstash-address")
    hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"type": "my-app"}))
    log.Hooks.Add(hook)


    log.Info("Hello World!")

}

Answer 2

Thanks for reply.

After a while looking for the solution to the problem, I found it!

Firstly, I needed to mount the file that is inside the instance in the docker-host.

To do this I add a Json file in the root folder of my project called Dockerrun.aws.json ( http://docs.aws.amazon.com/es_es/elasticbeanstalk/latest/dg/create_deploy_docker_image.html#create_deploy_docker_image_dockerrun )

That is the file that declares the shared folder (volumes) (beetwen docker-host and instance) where I save my log file . This line is equivalent to adding -v flag in the docker run command (https://docs.docker.com/engine/tutorials/dockervolumes/#mount-a-host-directory-as-data-volume). I do this this way because I can not add mount to a running instance and i cant stop it by ssh.

{
  "AWSEBDockerrunVersion": "1",
  "Volumes": [
  {
    "HostDirectory": "/var/log/",
    "ContainerDirectory": "/go/src/app/log"
  }
  ]
}

Then to tell aws that I want to download my log file when I request records. (Tail (last 100 lines), bundle or rotate) I add these files to the .ebextension folder in my project directory. ( http://docs.aws.amazon.com/en_us/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-extend )

Log_bundle.conf

Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_bundle.conf":
    Mode: "000755"
    Owner: root
    Group: root
    Content: |
      /var/log/application.log

Log_rotate.config

Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_rotate.conf":
    Mode: "000755"
    Owner: root
    Group: root
    Content: |
      /var/log/application.log

Log_tail.config

Files:
"/opt/elasticbeanstalk/tasks/publishlogs.d/log_tail.conf":
    Mode: "000755"
    Owner: root
    Group: root
    Content: |
      /var/log/application.log

Finally, I dont try Amazon Could Watch but is the next step.

Regards

Answer 3

If you use AWS, i would suggest to send Logs direct to AWS CloudWatch.

First create a new Log-Group in AWS Cloudwatch, for example "Production". In your Docker-Compose.yml (or via docker run..) add the AWS Logdriver:

    logging:
    driver: "awslogs"
    options:
      awslogs-region: "eu-central-1"
      awslogs-group: "Production"
      awslogs-stream: "MyApp"

Next creat a IAM user with Access to AWS Cloudwatch and add to the Dockerhost the credentials.

Example IAM Policy:

"Version" "2012-10-17" 
"Statement" 

"Action" "logs:CreateLogStream" "logs:PutLogEvents" "Effect" "Allow" "Resource"

On Ubuntu with systemd:

"Version" "2012-10-17" 
"Statement" 
  "Action" 
    "logs:CreateLogStream" 
    "logs:PutLogEvents" 
  "Effect" 
  "Allow" "Resource"

And add to the File:

[Service] Environment"AWS_ACCESS_KEY_ID=<aws_access_key_id>" 
Environment"AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>"

Run:

systemctl daemon-reload
service docker restart

Now your logs should appear in AWS Cloudwatch.

Answer 4

You could mount the log file from your container to your EC2 host. You can do this by using the -v flag when running your container:

docker run -v /var/log/my_host_log_file.log:/var/log/your_container_log_file.log your-image

Alternatively, you can configure your app to log to stdout and use syslog as your log driver (using the --log-driver=syslog switch). Your container logs will then be written to /var/log/messages on your host.