CODEWITHSUNDEEP
terraformaws-mskkafka-topic
Homer
Homer

Reputation: 137

Create MSK Kafka topic through Terraform

I am trying to create topics through Terraform, in a AWS MSK Kafka cluster, using the Mongey/kafka provider, but I always get connection refused.

I have the cluster configured as follows:

resource "aws_msk_cluster" "this" {

  cluster_name           = local.cluster_name
  kafka_version          = var.kafka_version # 2.8.2.tiered
  number_of_broker_nodes = var.number_of_broker_nodes
  enhanced_monitoring    = var.enhanced_monitoring
  storage_mode           = var.cluster_storage_mode
  tags                   = var.tags


  broker_node_group_info {
    client_subnets  = var.broker_node_client_subnets
    instance_type   = var.broker_node_instance_type
    security_groups = var.security_groups_id
  }
}

So I got the bootstrap brokers using

aws kafka get-bootstrap-brokers --cluster-arn <arn>

And got

{
    "BootstrapBrokerStringTls": "....kafka.eu-central-1.amazonaws.com:9094,....kafka.eu-central-1.amazonaws.com:9094,....kafka.eu-central-1.amazonaws.com:9094"
}

So I used them with the Mongey/kafka provider:

terraform {
  required_providers {
    kafka = {
      source = "Mongey/kafka"
    }
  }
}

provider "kafka" {
  bootstrap_servers = [
    "....kafka.eu-central-1.amazonaws.com:9094",
    "....kafka.eu-central-1.amazonaws.com:9094",
    "....kafka.eu-central-1.amazonaws.com:9094"
  ]
  tls_enabled = true
}


resource "kafka_topic" "topics" {
  for_each = var.kafka_topics

  name               = var.kafka_topics[each.key]
  replication_factor = 2
  partitions         = 100

  config = {
    "segment.ms"     = "20000"
    "cleanup.policy" = "compact"
  }
}

But I get

kafka: client has run out of available brokers to talk to: 3 errors occurred:
        connect: connection refused

Some configuration missing? Or maybe I should use another provider? I also saw the official documentation https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html but I would like to avoid any manual steps, and try to do it in Terraform.

Upvotes: 1

Views: 1831

Answers (1)

signaleleven
signaleleven

Reputation: 1

I used the Mongey/kafka provider in the past and your code looks ok on first glance, but I cannot see the subnets and the security groups you assigned to the broker (I see you set them with a variable that I don't see).

Make sure that the machine where you run terraform can connect to the brokers.

Upvotes: 0

Related Questions