CODEWITHSUNDEEP
amazon-web-servicesaws-cloudformationamazon-rdsamazon-aurora
Joey Yi Zhao
Joey Yi Zhao

Reputation: 42426

Failed to deploy `ProxyTargetGroup` for RDS Aurora postgresql

I have deployed an Aurora cluster and a db instance (postgresql 11.8) and a proxy as AWS. When I try to add a proxy target group like below code, the deploy takes one hour and timed out in 2 hours. I have attached the screenshot. It works fine if I manually add the target group through AWS console. I wonder what wrong with my configuration?

ProxyTargetGroup:
    Type: AWS::RDS::DBProxyTargetGroup
    Properties:
      DBProxyName: !Ref DBProxy
      DBClusterIdentifiers: [!Ref AuroraDBCluster]
      TargetGroupName: default
      ConnectionPoolConfigurationInfo:
          MaxConnectionsPercent: 100
          MaxIdleConnectionsPercent: 50
          ConnectionBorrowTimeout: 120

enter image description here

DBProxy:
    Type: AWS::RDS::DBProxy
    Properties: 
      Auth:
        - {AuthScheme: SECRETS, SecretArn: !Ref DBSecret, IAMAuth: REQUIRED}
      DBProxyName: ${self:provider.stackName}-dbproxy 
      DebugLogging: true
      EngineFamily: POSTGRESQL
      IdleClientTimeout: 30
      RequireTLS: true
      RoleArn: !GetAtt DBProxyRole.Arn
      VpcSecurityGroupIds:
        - !Ref ClusterSecurityGroup
      VpcSubnetIds:
        - !Ref SubnetAPublic
        - !Ref SubnetAPrivate
        - !Ref SubnetBPrivate
        - !Ref SubnetCPrivate
DBProxyRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: ${self:provider.stackName}-dbproxyRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - rds.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: ${self:provider.stackName}-dbproxyPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - secretsmanager:GetSecretValue
                  - secretsmanager:GetResourcePolicy
                  - secretsmanager:DescribeSecret
                  - secretsmanager:ListSecretVersionIds
                Resource:
                  - "arn:aws:secretsmanager:${opt:region}:${self:provider.accountId}:secret:${opt:stage}/${self:service.name}/AuroraUserSecret"

              - Effect: Allow
                Action:
                  - kms:*
                Resource: 'arn:aws:kms:${opt:region}:${self:provider.accountId}:key/*'
ClusterSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow traffic to client host
      VpcId:
        Ref: VPC
      SecurityGroupIngress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0

When the cloudformation is showing update in progress, I can see that the target group is added and available. But the cloudformation keeps showing in progress until timed out.

Upvotes: 2

Views: 3005

Answers (5)

klassikcat
klassikcat

Reputation: 1

For me, authrication failure was the key of the issue. Since IAM Role does not allow Proxy to access RDS and SecretsManager, it takes infinity time to form proxy target group, resulting in timeout error.

The correct IAM role should look like below. (Note that wildcard resource is discouraged by amazon due to the security issue)

  DBProxyPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: DBProxyPolicy
      Roles:
        - !Ref DBProxyRole
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - rds-db:connect
              - rds:DescribeDBProxies
              - rds:DescribeDBProxyTargetGroups
              - rds:RegisterDBProxyTargets
              - rds:DeregisterDBProxyTargets
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
              - secretsmanager:ListSecrets
            Resource: '*'

and be sure aws rds describe-db-proxy-targets --db-proxy-name ${yourProxyName} be look like this.

{
    "Targets": [
        {
            "Endpoint": "endpoint.url.region.rds.amazonaws.com",
            "TrackedClusterId": "${clusterIdHere}",
            "RdsResourceId": "${rdsResourceIdHere}",
            "Port": $portNumHere,
            "Type": "RDS_INSTANCE",
            "Role": "${Role}",
            "TargetHealth": {
                "State": "AVAILABLE"
            }
        },
        ...
    ]
}

not this.

{
    "Targets": [
        {
            "Endpoint": "endpoint.url.region.rds.amazonaws.com",
            "TrackedClusterId": "${clusterIdHere}",
            "RdsResourceId": "${rdsResourceIdHere}",
            "Port": $portNumHere,
            "Type": "RDS_INSTANCE",
            "Role": "${Role}",
            "TargetHealth": {
            "TargetHealth": {
                "State": "UNAVAILABLE",
                "Reason": "AUTH_FAILURE",
                "Description": "Proxy does not have any registered credentials"
            }
        },
        ...
    ]
}

Upvotes: 0

Komal Singh Rathore
Komal Singh Rathore

Reputation: 1

Please give Secret Manager full access and use depends-on condition for RDS-DBInstance in RDS Proxy and Proxy-target-groups and Proxy-Endpoint then it will work. If you have any doubt then please run this command in the AWs Cloud shell. It will show you the target group's health status and whether it is available or not

aws rds describe-db-proxy-targets --db-proxy-name $DB_PROXY_NAME.

Note: If you need a full template then ping me on this post :)

https://aws.amazon.com/getting-started/hands-on/set-up-shared-database-connection-amazon-rds-proxy/

Upvotes: 0

Zogrof
Zogrof

Reputation: 661

I had the same problem of CF stack being stuck in UPDATE_IN_PROGRESS for 2 hours. My issue was that I didn't specify the CidrIp property on the associated ingress security group rule. Docs say it's not required, but it is. The rule was never created and CF did't notify me.

Creating the ingress rule resolved the DBProxyTargetGroup issue for me.

Upvotes: 3

Jegatheesh Selvam
Jegatheesh Selvam

Reputation: 9

I have tried your code but the same "Error" will come

I have to change the small change "EngineFamily: POSTGRESQL" to "EngineFamily: Mysql"

Upvotes: 0

Marcin
Marcin

Reputation: 238081

I tried to recreate the issues using my own Aurora cluster. I had to fill out a lot of blanks as there are only few bits and pieces provided in the question.

Havever, I had no problems creating proxy with the fixed role. The full template that I've used is below:


Parameters:

  AuroraDBCluster:
    Type: String
    Default: database-22

  DBSecret:
    Type: String
    Default: arn:aws:secretsmanager:us-east-1:xxxxxxx:secret:postgres-wCBBqC   

  ClusterSecurityGroup:
    Type: AWS::EC2::SecurityGroup::Id
    Default: sg-0f52f72631fa40b56

  SubnetAPublic:
    Type: AWS::EC2::Subnet::Id

  SubnetAPrivate:
    Type: AWS::EC2::Subnet::Id

  SubnetBPrivate:
    Type: AWS::EC2::Subnet::Id

  SubnetCPrivate:
    Type: AWS::EC2::Subnet::Id


Resources:

  ProxyTargetGroup:
    Type: AWS::RDS::DBProxyTargetGroup
    Properties:
      DBProxyName: !Ref DBProxy
      DBClusterIdentifiers: [!Ref AuroraDBCluster]
      TargetGroupName: default
      ConnectionPoolConfigurationInfo:
          MaxConnectionsPercent: 100
          MaxIdleConnectionsPercent: 50
          ConnectionBorrowTimeout: 120


  DBProxy:
    Type: AWS::RDS::DBProxy
    Properties: 
      Auth:
        - {AuthScheme: SECRETS, SecretArn: !Ref DBSecret, IAMAuth: DISABLED}
      DBProxyName: ggggg-dbproxy 
      DebugLogging: true
      EngineFamily: POSTGRESQL
      IdleClientTimeout: 30
      RequireTLS: true
      RoleArn: !GetAtt DBProxyRole.Arn
      VpcSecurityGroupIds:
        - !Ref ClusterSecurityGroup
      VpcSubnetIds:
        - !Ref SubnetAPublic
        - !Ref SubnetAPrivate
        - !Ref SubnetBPrivate
        - !Ref SubnetCPrivate

  DBProxyRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: dbproxyRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - rds.amazonaws.com
            Action: sts:AssumeRole
      Policies:
          - PolicyName: AccessSecretAndKMS
            PolicyDocument: !Sub |
              {
                  "Version": "2012-10-17",
                  "Statement": [
                      {
                          "Sid": "VisualEditor0",
                          "Effect": "Allow",
                          "Action": "secretsmanager:GetSecretValue",
                          "Resource": "${DBSecret}"
                      },
                      {
                          "Sid": "VisualEditor1",
                          "Effect": "Allow",
                          "Action": "kms:Decrypt",
                          "Resource": "*",
                          "Condition": {
                              "StringEquals": {
                                  "kms:ViaService": "secretsmanager.${AWS::Region}.amazonaws.com"
                              }
                          }
                      }
                  ]
              }

Upvotes: 1

Related Questions