CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2aws-auto-scaling
Abhinav
Abhinav

Reputation: 195

EC2 - Auto Scaling Group - Termination Reason: Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state

I was trying to setup an ASG in my current AWS architecture, I did configure all the steps correctly, but the ASG does not scale as expected. I was seeing new instances in terminated state in my EC2 console. When I look at the activity logs, I found this message:

Launching a new EC2 instance: i-xxxxxxxxxxxx. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state.

Tried looking up what it means but I was not successful. GPT says that it relates to a permissions error on the key that I am using. I created a customer managed key to encrypt my instance's EBS volumes. Here is its policy:

{
    "Id": "key-consolepolicy-3",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::303567773654:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow access for Key Administrators",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::303567773654:role/aws-service-role/elasticache.amazonaws.com/AWSServiceRoleForElastiCache",
                    "arn:aws:iam::303567773654:role/aws-service-role/events.amazonaws.com/AWSServiceRoleForCloudWatchEvents",
                    "arn:aws:iam::303567773654:role/c95174a2135306l4753725t1w30356777365-LambdaSLRRole-1P6X3H0TXRS9T",
                    "arn:aws:iam::303567773654:role/aws-service-role/organizations.amazonaws.com/AWSServiceRoleForOrganizations",
                    "arn:aws:iam::303567773654:role/aws-service-role/trustedadvisor.amazonaws.com/AWSServiceRoleForTrustedAdvisor",
                    "arn:aws:iam::303567773654:role/aws-service-role/support.amazonaws.com/AWSServiceRoleForSupport",
                    "arn:aws:iam::303567773654:role/EMR_EC2_DefaultRole",
                    "arn:aws:iam::303567773654:role/aws-service-role/cloud9.amazonaws.com/AWSServiceRoleForAWSCloud9",
                    "arn:aws:iam::303567773654:role/EMR_AutoScaling_DefaultRole",
                    "arn:aws:iam::303567773654:role/EMR_DefaultRole"
                ]
            },
            "Action": [
                "kms:Create*",
                "kms:Describe*",
                "kms:Enable*",
                "kms:List*",
                "kms:Put*",
                "kms:Update*",
                "kms:Revoke*",
                "kms:Disable*",
                "kms:Get*",
                "kms:Delete*",
                "kms:TagResource",
                "kms:UntagResource",
                "kms:ScheduleKeyDeletion",
                "kms:CancelKeyDeletion"
            ],
            "Resource": "*"
        }
    ]
}

Seems like the policy is configured properly. Am I missing something here?

Upvotes: 4

Views: 18004

Answers (3)

skumar
skumar

Reputation: 1

Initially, I encrypted the AMI using a different KMS key, and the AMI was shared across accounts as part of the EC2 builder setup.

Subsequently, when I implemented a multi-region setup within the same AWS account, I shared the AMI across regions by creating a new multi-region KMS key. However, the Auto Scaling Group (ASG) was unable to launch the EC2 instances, and I encountered the same error mentioned earlier. Referred AWS post here

To resolve this, I attached the service-linked role to the newly created KMS key, which successfully addressed the issue and enabled the ASG to launch the instances.

Upvotes: 0

DGM
DGM

Reputation: 26979

Somehow I had this error even when my Launch template did not specify any encryption. :( That said I did need to enable encryption, which necessitated this fix anyway.

As @jordanm said above, the AWSServiceRoleForAutoScaling role needs to have access to the keys, especially customer managed KMS keys.

https://docs.aws.amazon.com/autoscaling/ec2/userguide/key-policy-requirements-EBS-encryption.html

That link provided the following example to be placed in the KMS key policy section (not in IAM)

{
   "Sid": "Allow service-linked role use of the customer managed key",
   "Effect": "Allow",
   "Principal": {
       "AWS": [
           "arn:aws:iam::account-id:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
       ]
   },
   "Action": [
       "kms:Encrypt",
       "kms:Decrypt",
       "kms:ReEncrypt*",
       "kms:GenerateDataKey*",
       "kms:DescribeKey"
   ],
   "Resource": "*"
}

{
   "Sid": "Allow attachment of persistent resources",
   "Effect": "Allow",
   "Principal": {
       "AWS": [
           "arn:aws:iam::account-id:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
       ]
   },
   "Action": [
       "kms:CreateGrant"
   ],
   "Resource": "*",
   "Condition": {
       "Bool": {
           "kms:GrantIsForAWSResource": true
       }
    }
}

Upvotes: 5

Johnny Chu
Johnny Chu

Reputation: 929

We ran into the same problem. Is the KMS key shared in multiple accounts? Did you try to disable KMS when you created the AMI? Make you you're using the same key when you create the AMI.

Upvotes: 0

Related Questions