CODEWITHSUNDEEP
node.jsamazon-web-servicesamazon-ecsaws-cdkaws-fargate
Yusuke Niwa
Yusuke Niwa

Reputation: 51

How do I solve "ResourceInitializationError" error for Task execution in ECS on Fargate?

What I want to do

I want to create Node.js (built with Nest.js) server in the infrastructure as follows:

infra-structure-image

GitHub repo is here.

Notice:

Current behavior

When I deploy the AWS CloudFormation stack with cdk deploy, it is stuck in the ECS service creation at CREATE_IN_PROGRESS state. I can see ECS task execution error logs in ECS management console as follows:

STOPPED (ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr.ap-northeast-1.amazonaws.com/: dial tcp 99.77.62.61:443: i/o timeout)

If I don't delete stack or set minimum number of task to 0, ECS service continuously try to execute tasks for hours and finally get timeout error.

I have already checked some points based on this official article.

And I have checked 'hello world' with this docker image in local machine.

Reproduction Steps

A minimal GitHub repo is here.

$ git clone https://github.com/Fanta335/cdk-ecs-nest-app
$ cd cdk-ecs-nest-app
$ npm install

AWS CDK toolkit is used in this project, so you need to run npm install -g aws-cdk if you have not installed AWS CDK toolkit in your local machine.

And if you have not set default IAM user configuration in aws cli, you need to run aws configure in order to pass environment variables to the CloudFormation stack.

$ cdk deploy

Then the deployment should be stuck.

Versions

Upvotes: 2

Views: 1324

Answers (1)

Yusuke Niwa
Yusuke Niwa

Reputation: 51

The problem was DNS resolution has not been enabled in ECR VPC endpoints. I should have set privateDnsEnabled: true manually to the InterfaceVpcEndpoint instances in cdk-ecs-nest-app-stack.ts file as follows:

    const ECSPrivateLinkAPI = new ec2.InterfaceVpcEndpoint(this, "ECSPrivateLinkAPI", {
      vpc,
      service: new ec2.InterfaceVpcEndpointService(`com.amazonaws.${REGION}.ecr.api`),
      securityGroups: [securityGroupPrivateLink],
      privateDnsEnabled: true, // HERE
    });
    const ECSPrivateLinkDKR = new ec2.InterfaceVpcEndpoint(this, "ECSPrivateLinkDKR", {
      vpc,
      service: new ec2.InterfaceVpcEndpointService(`com.amazonaws.${REGION}.ecr.dkr`),
      securityGroups: [securityGroupPrivateLink],
      privateDnsEnabled: true, // HERE
    });

According to the CDK docs, the default value of privateDnsEnabled is defined by the service which uses this VPC endpoint.

privateDnsEnabled?
Type: boolean (optional, default: set by the instance of IInterfaceVpcEndpointService, or true if not defined by the instance of IInterfaceVpcEndpointService)

I didn't checked the default privateDnsEnabled values of com.amazonaws.${REGION}.ecr.api and com.amazonaws.${REGION}.ecr.dkr but we have to set true manually in CDK Toolkit.

Upvotes: 3

Related Questions