unable to pull secrets or registry auth: pull command failed: : signal: killed

Question

I have spent the entire day trying to deploy an image to Fargate but I keep running into the following error:

ResourceInitializationError: unable to pull secrets or registry auth: pull command failed: : signal: killed

I am using the following cloudformation template (abridged):

Parameters:
  VPC:
    Type: AWS::EC2::VPC::Id
    Description: VPC Id. VPC's private subnets must have a NAT Gateway to download container image.
  PublicSubnets:
    Description: Choose which public subnets the Load Balancer and ECS Service should be deployed to
    Type: List
  IamCertificateArn:
    Type: String
    Description: The IAM Certificate Arn, which must exist in the same region
  Image:
    Type: String
    Description: Docker image. You can use images in the Docker Hub registry or specify other repositories (repository-url/image:tag).
  ContainerPort:
    Type: Number
    Description: Port on which the application listens within the docker container
    Default: 80
  LoadBalancerPort:
    Type: Number
    Default: 443
  RDSSecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
    Description: Security Group of the RDS Instance
  HealthCheckPath:
    Type: String
    Description: The health check path starting with / e.g. /health
  HostedZoneName:
    Type: String
    Description: Domain name for your website (example.com.). Note it must end in a .
  HostedZoneID:
    Type: String
    Description: Hosted Zone Id

Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    DependsOn: 
      - LogGroup
      - TaskExecutionRole
    Properties:
      Family: !Join ["", [!Ref "AWS::StackName", TaskDefinition]]
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      Cpu: 512
      Memory: 1GB
      ExecutionRoleArn: !Ref TaskExecutionRole
      TaskRoleArn: !Ref TaskExecutionRole
      ContainerDefinitions:
        - Name: !Ref "AWS::StackName"
          Image: !Ref Image
          PortMappings:
            - ContainerPort: !Ref ContainerPort
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref LogGroup
              awslogs-stream-prefix:
                !Join ["", [!Ref "AWS::StackName", -ecs-app]]
          Command: ["./app -f config.toml"]

  Cluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Join ["", [!Ref "AWS::StackName", "-", "Cluster"]]

  Service:
    Type: AWS::ECS::Service
    DependsOn:
      - ListenerHTTPS
    Properties:
      ServiceName: !Join ["", [!Ref "AWS::StackName", "-", "Service"]]
      Cluster: !Ref Cluster
      TaskDefinition: !Ref TaskDefinition
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      DesiredCount: 1
      HealthCheckGracePeriodSeconds: 30
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          Subnets: !Ref PublicSubnets
          SecurityGroups:
            - !Ref ContainerSecurityGroup
      LoadBalancers:
        - ContainerName: !Ref AWS::StackName
          ContainerPort: !Ref ContainerPort
          TargetGroupArn: !Ref TargetGroup

  # API Load Balancer

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: !Ref HealthCheckPath
      HealthCheckTimeoutSeconds: 10
      UnhealthyThresholdCount: 2
      HealthyThresholdCount: 2
      Name: !Join ["", [!Ref AWS::StackName, "-", "TrgtGrp"]]
      Port: !Ref ContainerPort
      Protocol: HTTP
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 60 # default is 300
      TargetType: ip
      VpcId: !Ref VPC

  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Join ["", [!Ref AWS::StackName, "-", "ELB"]]
      Scheme: internet-facing
      SecurityGroups: 
        - !Ref LoadBalancerSecurityGroup
      Subnets: !Ref PublicSubnets

  ListenerHTTPS:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - TargetGroupArn: !Ref TargetGroup
          Type: forward
      LoadBalancerArn: !Ref LoadBalancer
      Port: !Ref LoadBalancerPort
      Protocol: HTTPS
      Certificates:
        - CertificateArn: !Ref IamCertificateArn
  
  TaskExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
              - ec2.amazonaws.com
              - ecs.amazonaws.com
              - ecs-tasks.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyDocument: {
            "Version": "2012-10-17",
            "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "s3:GetObject"
              ],
              "Resource": [
                "arn:aws:s3:::config.toml"
              ]
            }
            ]
          }
          PolicyName: !Join ["", [!Ref "AWS::StackName", "-", "DownloadConfigFromS3Policy"]]
        - PolicyDocument: {
            "Version": "2012-10-17",
            "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "cr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage"
              ],
              "Resource": "*"
            }
            ]
          }
          PolicyName: !Join ["", [!Ref "AWS::StackName", "-", "PullImagePolicy"]]
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
        - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role


  VPCEndpointECRAPI:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      SecurityGroupIds:
        - !Ref PrivateLinkSecurityGroup
      ServiceName: !Join ["", ["com.amazonaws.", !Ref "AWS::Region", ".ecr.api"]]
      SubnetIds: !Ref PublicSubnets
      VpcId: !Ref VPC

  VPCEndpointECRDKR:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      SecurityGroupIds:
        - !Ref PrivateLinkSecurityGroup
      ServiceName: !Join ["", ["com.amazonaws.", !Ref "AWS::Region", ".ecr.dkr"]]
      SubnetIds: !Ref PublicSubnets
      VpcId: !Ref VPC

  VPCEndpointLogs:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      SecurityGroupIds:
        - !Ref PrivateLinkSecurityGroup
      ServiceName: !Join ["", ["com.amazonaws.", !Ref "AWS::Region", ".logs"]]
      SubnetIds: !Ref PublicSubnets
      VpcId: !Ref VPC

  VPCEndpointS3:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      SecurityGroupIds:
        - !Ref PrivateLinkSecurityGroup
      ServiceName: !Join ["", ["com.amazonaws.", !Ref "AWS::Region", ".s3"]]
      SubnetIds: !Ref PublicSubnets
      VpcId: !Ref VPC
  
  VPCEndpointSSM:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      SecurityGroupIds:
        - !Ref PrivateLinkSecurityGroup
      ServiceName: !Join ["", ["com.amazonaws.", !Ref "AWS::Region", ".ssm"]]
      SubnetIds: !Ref PublicSubnets
      VpcId: !Ref VPC

  # Security Groups

  LoadBalancerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Load balancer security group
      GroupName: !Join ["", [!Ref "AWS::StackName", "-", "LoadBalancerSecurityGroup"]]
      Tags:
        - Key: ProjectName
          Value: !Ref "AWS::StackName"
      VpcId: !Ref VPC

  LoadBalancerInboundRuleHTTP:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref LoadBalancerSecurityGroup
      IpProtocol: tcp
      FromPort: 80
      ToPort: 80
      CidrIp: 0.0.0.0/0
  
  LoadBalancerOutboundRuleHTTP:
    Type: AWS::EC2::SecurityGroupEgress
    Properties:
      GroupId: !Ref LoadBalancerSecurityGroup
      IpProtocol: tcp
      FromPort: 80
      ToPort: 80
      CidrIp: 0.0.0.0/0
  
  LoadBalancerInboundRuleHTTPS:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref LoadBalancerSecurityGroup
      IpProtocol: tcp
      FromPort: 443
      ToPort: 443
      CidrIp: 0.0.0.0/0
  
  LoadBalancerOutboundRuleHTTPS:
    Type: AWS::EC2::SecurityGroupEgress
    Properties:
      GroupId: !Ref LoadBalancerSecurityGroup
      IpProtocol: tcp
      FromPort: 443
      ToPort: 443
      CidrIp: 0.0.0.0/0
  
  LoadBalancerInboundRuleContainer:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref LoadBalancerSecurityGroup
      IpProtocol: tcp
      FromPort: !Ref ContainerPort
      ToPort: !Ref ContainerPort
      SourceSecurityGroupId: !GetAtt ContainerSecurityGroup.GroupId
  
  LoadBalancerOutboundRuleContainer:
    Type: AWS::EC2::SecurityGroupEgress
    Properties:
      GroupId: !Ref LoadBalancerSecurityGroup
      IpProtocol: tcp
      FromPort: !Ref ContainerPort
      ToPort: !Ref ContainerPort
      DestinationSecurityGroupId: !GetAtt ContainerSecurityGroup.GroupId

  ContainerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Accepts traffic from and to load balancer
      GroupName: !Join ["", [!Ref "AWS::StackName", "-", "ContainerSecurityGroup"]]
      Tags:
        - Key: ProjectName
          Value: !Ref "AWS::StackName"
      VpcId: !Ref VPC
      
  ContainerInboundRuleHTTP:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref ContainerSecurityGroup
      IpProtocol: tcp
      FromPort: !Ref ContainerPort
      ToPort: !Ref ContainerPort
      SourceSecurityGroupId: !GetAtt LoadBalancerSecurityGroup.GroupId
  
  ContainerOutboundRuleHTTP:
    Type: AWS::EC2::SecurityGroupEgress
    Properties:
      GroupId: !Ref ContainerSecurityGroup
      IpProtocol: tcp
      FromPort: !Ref ContainerPort
      ToPort: !Ref ContainerPort
      DestinationSecurityGroupId: !GetAtt LoadBalancerSecurityGroup.GroupId

  ContainerInboundRuleRDS:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref ContainerSecurityGroup
      IpProtocol: tcp
      FromPort: 5432
      ToPort: 5432
      SourceSecurityGroupId: !Ref RDSSecurityGroupId
    
  ContainerInboundRuleRDS:
    Type: AWS::EC2::SecurityGroupEgress
    Properties:
      GroupId: !Ref ContainerSecurityGroup
      IpProtocol: tcp
      FromPort: 5432
      ToPort: 5432
      DestinationSecurityGroupId: !Ref RDSSecurityGroupId
  
  PrivateLinkSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties: 
      GroupName: PrivateLink-SecurityGroup
      GroupDescription: PrivateLink-SecurityGroup
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0 #should be cidrip of vpc
      VpcId: !Ref VPC

I have reviewed several questions in StackOverflow and in aws forums. I can confirm:

the VPC has DNS enabled
Assign Public IPs is enabled on Fargate
There are no Private Subnets being used
The ECR Repository is private
I'm not storing any secrets in secretmanager

I have tried all kinds of TaskExecutionPolicies granting full access to ECS, but I still get the same error.

I honestly don't know what else to try. Would really appreciate some help on this one.

Marcin · Accepted Answer

Your issue is due to incorrect ContainerSecurityGroup. It only allows outbound connections to your ALB. Thus it can't fetch your images. I would add the following rule:

  ContainerOutboundAll:
    Type: AWS::EC2::SecurityGroupEgress
    Properties:
      GroupId: !Ref ContainerSecurityGroup
      IpProtocol: tcp
      FromPort: 0
      ToPort: 65535
      CidrIp: 0.0.0.0/0

This will allow for all outbound traffic, inducing the done to ECR. You can further play around with it allowing access to only VPC interface endpoints if you want.

Please note: The ContainerOutboundAll should address your current error. Your template is long and complex, and can have more issues, which are not apparent yet. If so, new question would be better for the those issues (if any).

unable to pull secrets or registry auth: pull command failed: : signal: killed

Answers (1)

Related Questions