CODEWITHSUNDEEP
amazon-web-servicesaws-cdkamazon-kinesis-firehoseopensearch
Amor.o
Amor.o

Reputation: 540

IAM has no access to Elasticsearch domain while Setting Opensearch as a Firehose destination

I'm trying to build a firehose that delivers data to Opensearch using CDK with Python, when I do that from Console it works like a charm however when I try to deploy it with CDK it gives me this error

Resource handler returned message: "Verify that the IAM role has access to the ElasticSearch domain.

It is deriving me crazy, I tried every IAM es policy but no one works. this is my configuration here I define the role

        self.firehose_role = Role(
            self,
            "FirehoseRole",
            role_name=f"{construct_name}",
            assumed_by=ServicePrincipal(service="firehose.amazonaws.com"),
        )

        self.firehose_role.add_to_policy(
            PolicyStatement(
                actions=["es:*"],
                resources=[
                    "arn:aws:es:eu-west-1:xxx:domain/my-domain",
                    "arn:aws:es:eu-west-1:xxx:domain/my-domain/*",
                ],
            ),
        )

        self.firehose_role.add_to_policy(
            PolicyStatement(
                actions=[
                    "ec2:DescribeVpcs",
                    "ec2:DescribeVpcAttribute",
                    "ec2:DescribeSubnets",
                    "ec2:DescribeSecurityGroups",
                    "ec2:DescribeNetworkInterfaces",
                    "ec2:CreateNetworkInterface",
                    "ec2:CreateNetworkInterfacePermission",
                    "ec2:DeleteNetworkInterface",
                ],
                resources=[
                    "*",
                ],
            ),
        )

Then setting OS configuration as following

        os_config =CfnDeliveryStream.AmazonopensearchserviceDestinationConfigurationProperty(
            index_name="xxx",
            role_arn=self.firehose_role.role_arn,
            s3_configuration=CfnDeliveryStream.S3DestinationConfigurationProperty(
                #bucket config
            ),
            buffering_hints=CfnDeliveryStream.ElasticsearchBufferingHintsProperty(
                interval_in_seconds=120,
                size_in_m_bs=5,
            ),
            cloud_watch_logging_options=CfnDeliveryStream.CloudWatchLoggingOptionsProperty(
                enabled=True,
                log_group_name=log_group.log_group_name,
                log_stream_name=log_stream.log_stream_name,
            ),
            domain_arn="arn:aws:es:eu-west-1:xxx:domain/my-domain",
            retry_options=CfnDeliveryStream.AmazonopensearchserviceRetryOptionsProperty(
                duration_in_seconds=180
            ),
            s3_backup_mode="AllDocuments",
            vpc_configuration=CfnDeliveryStream.VpcConfigurationProperty(
                role_arn=self.firehose_role.role_arn,
                security_group_ids=["xxx"],
                subnet_ids=["xxx", "xxx"],
            ),
        )

To finally setting Kinesis Delivery stream

        self.delivery_stream = CfnDeliveryStream(
            self,
            id="FirehoseDS",
            delivery_stream_name=f"{construct_name}-ds",
            delivery_stream_type="DirectPut",
            amazonopensearchservice_destination_configuration=os_config,
        )

I run out of ideas, PS: knowing that OS access policy is wide open and I don't think the problem is there because if it was there at least I would've been able to deploy Firehose the I would have problems with data not reaching, but now I'm not able to even deploy Firehose

Upvotes: 1

Views: 1307

Answers (3)

Amor.o
Amor.o

Reputation: 540

Apparently I managed to solve it by deploying IAM actions separately as an IAM Managed Policy, then adding the policy to Firehose role. I'm not sure why but I think CDK was trying to create Firehose before the IAM is ready to use which throws an error and prevent it from being deployed.

Upvotes: 2

vumdao
vumdao

Reputation: 631

If you enable fine-grained access control in opensearch, the IAM role ARN need to be added to opensearch through master user or use opensearch securiry dashboard.

Upvotes: 0

lprentice
lprentice

Reputation: 204

The access policy on your Opensearch domain could be preventing access to the IAM role depending on how its configured. This will also need to allow the IAM role used by firehose to perform the same actions on the domain. https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html

Assuming a custom policy has been set on the domain the following statement would be added into the access policy to give access to the role being used by firehose.

{
  "Sid": "AllowFirehose",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::xxx:role/firehose-role"
    ]
  },
  "Action": "es:*",
  "Resource": [
    "arn:aws:es:eu-west-1:xxx:domain/my-domain",
    "arn:aws:es:eu-west-1:xxx:domain/my-domain/*"
  ]
}

Upvotes: 2

Related Questions