AWS Lambda gets 'Access Denied' error when calling Redshift 'CreateCluster' operation even when IAM role has full Redshift permission

Question

I have a Lambda function (rsops) written in Python 2.7 to create Redshift cluster by calling boto3's 'create_cluster()' method:

def spinup_cluster(CID, RSU, RSP, RSDB, RSSG, RSAZ, RSPG):
    RSC = boto3.client('redshift', region_name=RSAZ[:-1])

    return RSC.create_cluster(
        DBName=RSDB,
        ClusterIdentifier=CID,
        ClusterType='multi-node',
        NodeType='ds2.xlarge',
        MasterUsername=RSU,
        MasterUserPassword=RSP,
        VpcSecurityGroupIds=[RSSG],
        ClusterSubnetGroupName='data',
        AvailabilityZone=RSAZ,
        PreferredMaintenanceWindow='sun:03:00-sun:03:30',
        ClusterParameterGroupName=RSPG,
        AutomatedSnapshotRetentionPeriod=1,
        Port=5439,
        ClusterVersion='1.0',
        AllowVersionUpgrade=True,
        NumberOfNodes=2,
        PubliclyAccessible=True,
        Tags=[
            {
                'Key': 'product',
                'Value': 'data'
            },
        ],
        Encrypted=False)

The IAM role assigned to this Lambda function has full access to Redshift (for testing purpose):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LambdaInvokeLambda",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction"
            ],
            "Resource": [
                "arn:aws:lambda:us-east-1:012345678901:function:spinuprs*",
                "arn:aws:lambda:us-east-1:012345678901:function:rsops*"
            ]
        },
        {
            "Sid": "PassRoleOverToUser",
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "iam:PassRole"
            ],
            "Resource": [
                "arn:aws:lambda:us-east-1:012345678901:function:rsops*",
                "arn:aws:redshift:us-east-1:012345678901:cluster:*",
                "arn:aws:redshift:us-west-2:012345678901:cluster:*"
            ]
        },
        {
            "Sid": "RSAccess",
            "Action": "redshift:*",
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Trust policy grants 'AssumeRole' perm to EC2, Lambda and Redshift:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "ec2.amazonaws.com",
          "redshift.amazonaws.com",
          "lambda.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

But I keep getting 'Access Deinied' error says I don't have perm to make 'CreateCluster' call:

Spin up testcluster (us-east-1, 1-node) ...
Traceback (most recent call last):
  File "./rsops.py", line 163, in <module>
    resp=spinup_cluster(cid, rsu, rsp, rsdb, rssg, rsaz, rspg)
  File "./rsops.py", line 87, in spinup_cluster
    Encrypted=False)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 624, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.UnauthorizedOperation: An error occurred (UnauthorizedOperation) when calling the CreateCluster operation: Access Denied. Please ensure that your IAM Permissions allow this operation.

I'm able to call delete_cluster() and describe_clusters() without problem, I also called 'get_caller_identity()' from boto3 STS API to verify it is using the correct IAM role.

I'm running the Lambda function within a private subnet which has internet access through a NAT instance, but private subnet should not be an issue as I have other Lambda funcs running in it. I spinned up an instance with following environment in the same subnet with the same IAM role attached but I still get the same error:

AMI: amzn-ami-hvm-2018.03.0.20181129-x86_64-gp2
Python version: 2.7.14
boto3 version: 1.9.64

Please let me know if further details are needeed. I've been trying to debug this for a week but I couldn't figure it out, any help would be appreciated!

Answer 1

Just came back from a AWS loft nearby, a nice guy there helped me figure out I need a bunch of 'ec2:Describe*' perms in my IAM:

ec2:DescribeAccountAttributes
ec2:DescribeAddresses
ec2:DescribeAvailabilityZones
ec2:DescribeSecurityGroups
ec2:DescribeSubnets
ec2:DescribeVpcs
ec2:DescribeInternetGateways

I guess Lambda needs them to check if the VPC/Subnet/Security Group you specified in 'create_cluster' call actually exist. I actually modified the built-in 'AmazonRedshiftFullAccess' policy for my role to tight down permissions.