CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2amazon-ecs
Dan
Dan

Reputation: 3268

EC2 instance not joining ECS Cluster

I have an EC2 cluster and I'm trying to add EC2 machines to it.

I have added ECS_CLUSTER=cluster_name to /etc/ecs/ecs.config with user data but the instance doesn't join the cluster.

I'm looking for log files or any other evidence of the cluster-joining process that may indicate why my instance is not joining. I do not have a /var/log/ecs folder.

Upvotes: 12

Views: 16935

Answers (5)

Narayan Yerrabachu
Narayan Yerrabachu

Reputation: 1823

The ECS agent fails to start when installed via the user data script: This issue occurs when services are installed and started by the user data scripts, which depend on ecs.service with an "After" directive. The cloud-final.service waiting for the completion of the user data creates a circular dependency, preventing the system from loading.

    #!/bin/bash

    echo "ECS_CLUSTER=Cluster name" | sudo tee /etc/ecs/ecs.config
    sudo cp /usr/lib/systemd/system/ecs.service /etc/systemd/system/ecs.service
    sudo sed -i '/After=cloud-final.service/d' /etc/systemd/system/ecs.service
    sudo systemctl daemon-reload
    sudo systemctl stop ecs
    systemctl enable ecs
    sudo systemctl start ecs
    
    sudo systemctl status ecs

ref: ECS Agent fails to start

Upvotes: 0

Morgan W.
Morgan W.

Reputation: 11

Symptom of problem:

You have an ECS cluster with a capacity provider that is successfully provisioning EC2 instances. However, your ECS cluster shows "Container instances" as an empty list, indicating the provisioned EC2 instances are not being attached to the cluster. You have also made sure that your launch template has "user data" defined on it with a bash script similar to the following (replacing variables appropriately):

#!/bin/bash

## Configure cluster name using the template variable ${ecs_cluster_name}

echo ECS_CLUSTER='${ecs_cluster_name}' >> /etc/ecs/ecs.config

Reason for problem:

You have EC2s. The EC2s have a launch script. However, the EC2s are not using an AMI with an ECS agent installed and running that will interpret /etc/ecs/ecs.config and then attach itself to the appropriate cluster.

Solution:

  1. Use an ECS-optimized AMI for your launch definition instead of a regular AMI. Here is a list of recommended AMIs from Amazon: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html
  2. If you already had an EC2 running prior to the above change, then terminate the existing EC2; alternately go to your capacity provider and perform an "instance refresh" operation.
  3. Now observe that the "Container instances" list should no longer be empty.

Upvotes: 1

steven87vt
steven87vt

Reputation: 540

Same deal here except mine was related to a configuration option I found while setting up my first cluster. I specified propagate tags from the ec2 instance which I had found here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.html

# ecs.config
ECS_CONTAINER_INSTANCE_PROPAGATE_TAGS_FROM=ec2_instance`

The log output suggested my EC2 environment variables for ECS were asking for a new feature set: Error registering: InvalidParameterException: Long arn format must be enabled for tagging.

After some Googling I found the issue as described here: https://github.com/terraform-providers/terraform-provider-aws/issues/10762#issue-518401992

Solution

You need to either configure a specific user scope or preform the action as root.

  1. Go into your web console as root.
  2. Head over to ECS
  3. Select Account Settings in the left hand nav pane.
  4. Enable the newer arn format options. In my case I enabled all of them.

enter image description here

  1. Log out, sign in as a non root user and reboot your ec2 instances.

Upvotes: 0

Adiii
Adiii

Reputation: 60114

Restart ECS does not fix the issue.

In my case i check logs under

/var/logs/ecs

tail -f ecs-agent.log.2018-09-05-15

So I notice 

2018-09-05T15:26:22Z [ERROR] Could not register: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Just Assign ECS role and instance registered with the cluster.

Maybe this answer helps some.

Upvotes: 3

Daniel Scott
Daniel Scott

Reputation: 7971

You need to use an ECS AMI,

Or you need to install the ECS agent

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html

And then you need to start/restart the ecs agent after changing that config

restart ecs

Upvotes: 5

Related Questions