CODEWITHSUNDEEP
amazon-web-serviceslogstashamazon-ecs
Liav
Liav

Reputation: 155

How can I install aws cli, from WITHIN the ECS task?

Question: How can I install aws cli, from WITHIN the ECS task ?

DESCRIPTION: I'm using a docker container to run the logstash application (it is part of the elastic family).

The docker image name is "docker.elastic.co/logstash/logstash:7.10.2" This logstash application needs to write to S3, thus it needs AWS CLI installed. If aws is not installed, it crashes.

# STEP 1 # To avoid crashing, when I used this application only as a docker, I ran it in a way that I caused the 'logstash start' to be delayed, after docker container was started. I did this by adding "sleep" command to an external docker-entrypoint file, before it starts the logstash.

This is how it looks in the docker-entrypoint file: sleep 120

if [[ -z $1 ]] || [[ ${1:0:1} == '-' ]] ; then
  exec logstash "$@"
else
  exec "$@"
fi
 # EOF

# STEP 2 # run the docker with "--entrypoint" flag so it will use my entrypoint file

docker run \
           -d \
           --name my_logstash \
           -v /home/centos/DevOps/psifas_logstash_docker-entrypoint:/usr/local/bin/psifas_logstash_docker-entrypoint  \
           -v /home/centos/DevOps/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \
           -v /home/centos/DevOps/logstash.yml:/usr/share/logstash/config/logstash.yml \
           --entrypoint /usr/local/bin/psifas_logstash_docker-entrypoint  \
           docker.elastic.co/logstash/logstash:7.10.2

# STEP 3 # install aws cli and configure aws cli from the server hosting the docker:

docker exec -it -u root <DOCKER_CONTAINER_ID> yum install awscli -y
docker exec -it <DOCKER_CONTAINER_ID> aws configure set aws_access_key_id <MY_aws_access_key_id>
docker exec -it <DOCKER_CONTAINER_ID> aws configure set aws_secret_access_key <MY_aws_secret_access_key>
docker exec -it <DOCKER_CONTAINER_ID> aws configure set region <MY_region>

This worked for me, Now I want to "translate" this flow into an AWS ECS task. in ECS I will use parameters instead of running the above 3 "aws configure" commands.

MY QUESTION How can I do my 3rd step, installing aws cli, from WITHIN the ECS task ? (meaning not to run it on the EC2 server hosting the ECS cluster)

When I was working on the docker I also thought of these options to use the aws cli:

  1. find an official elastic docker image containing both logstash and aws cli. <-- I did not find one.
  2. create such an image by myself and use. <-- I prefer not , because I want to avoid the maintenance of creating new custom images when needed (e.g when new version of logstash image is available).

Eventually I choose the 3 steps above, but I'm open to suggestion.

Also, My tests showed that running 2 containers within the same ECS task:

  1. logstah
  2. awscli and then the logstash container will use the aws cli container (image "amazon/aws-cli") is not working.

THANKS A LOT IN ADVANCE :-)

Upvotes: 2

Views: 2251

Answers (2)

Liav
Liav

Reputation: 155

Mark B, your answer helped me to solve this. Thanks!

writing here the solution in case it will help somebody else.

There is no need to install AWS CLI, in the logstash docker container running inside the ECS task.

Inside the logstash container (from image "docker.elastic.co/logstash/logstash:7.10.2") there is AWS SDK to connect to the S3.

The only thing required is to allow the ECS Task execution role, access to S3. (I attached AmazonS3FullAccess policy)

Upvotes: 1

Mark B
Mark B

Reputation: 201078

Your option #2, create the image yourself, is really the best way to do this. Anything else is going to be a "hack". Also, you shouldn't be running aws configure for an image running in ECS, you should be assigning a IAM role to the task, and the AWS CLI will pick that up and use it.

Upvotes: 1

Related Questions