CODEWITHSUNDEEP
amazon-web-servicesapache-kafkaterraformaws-event-bridge
JPlanken
JPlanken

Reputation: 63

Required "source" variable for 'aws_pipes_pipe' terraform resource when using self managed Kafka

I’m trying to create an Event bridge pipe with a self-managed Kafka cluster as a source and a lambda function as a target through terraform. However, I’m encountering the following issue:

The terraform configuration expects a source; as can be seen in the documentation this is typically an ARN.

enter image description here

Reference: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/pipes_pipe

However, this cluster doesn't have an ARN as it is not managed by AWS but by Confluent. What do I fill in as the source in this case?

Example code below:

resource "aws_pipes_pipe" "eventbridge_northbound" {
  depends_on = [aws_iam_role_policy.target]
  name       = "my_eventbridge_pipe"
  role_arn   = aws_iam_role.my_eventbridge_role.arn
  source     = ???
  target     = aws_lambda_function.eventbridge_target.arn

  source_parameters {
    self_managed_kafka_parameters {

      additional_bootstrap_servers = [data.aws_secretsmanager_secret.confluent_bootstrap_server]
      topic_name = var.topicname

      credentials {
        basic_auth = data.aws_secretsmanager_secret.confluent_auth_credentials
      }
    }
  }
}

Upvotes: 0

Views: 181

Answers (1)

JPlanken
JPlanken

Reputation: 63

Instead of 'arn,' you can use 'smk', which stands for 'self-managed kafka', followed by the bootstrap server name. E.g. smk://pkc-p55xm.us-east-1.aws.confluent.cloud:9092

Your code example should then be something like this:

resource "aws_pipes_pipe" "eventbridge_northbound" {
  depends_on = [aws_iam_role_policy.target]
  name       = "my_eventbridge_pipe"
  role_arn   = aws_iam_role.my_eventbridge_role.arn
  source     = format("smk://%s", data.aws_secretsmanager_secret.confluent_bootstrap_server)
  target     = aws_lambda_function.eventbridge_target.arn

  source_parameters {
    self_managed_kafka_parameters {

      topic_name = var.topicname

      credentials {
        basic_auth = data.aws_secretsmanager_secret.confluent_auth_credentials
      }
    }
  }
}

Upvotes: 0

Related Questions