CODEWITHSUNDEEP
apache-kafka
CBP
CBP

Reputation: 799

Is it possible to create a topic with Kafka Rest Proxy?

I am trying to create a topic from Kafka Rest Proxy but I don't see any documentation for it. I'm hoping there is a way to do it so I don't need to programmatically create a topic differently than I do all other communication. Does anyone know if it possible?

I don't see any documentation for it here: http://docs.confluent.io/1.0/kafka-rest/docs/api.html.

Thanks for your help.

Upvotes: 6

Views: 14381

Answers (5)

jaiks
jaiks

Reputation: 526

It is possible with the latest version. See here: https://docs.confluent.io/platform/current/tutorials/examples/clients/docs/rest-proxy.html#basic-producer-and-consumer

From the docs:

KAFKA_CLUSTER_ID=$(docker-compose exec rest-proxy curl -X GET \
     "http://localhost:8082/v3/clusters/" | jq -r ".data[0].cluster_id")

Verify the parameter KAFKA_CLUSTER_ID has a valid value. For the example in this tutorial, it is shown as lkc-56ngz, but it will differ in your output.

echo $KAFKA_CLUSTER_ID
Create the Kafka topic test1 using the AdminClient functionality of the REST Proxy API v3.

docker-compose exec rest-proxy curl -X POST \
     -H "Content-Type: application/json" \
     -d "{\"topic_name\":\"test1\",\"partitions_count\":6,\"configs\":[]}" \
     "http://localhost:8082/v3/clusters/${KAFKA_CLUSTER_ID}/topics" | jq .

Upvotes: 3

Michael Heil
Michael Heil

Reputation: 18515

With the version 3 of the REST Proxy API it is now possible.

According to the Confluent REST Proxy API Reference the creation of a topic is possible with the REST Proxy API v3 that is currently available as a preview feature.

An example of a topic creation request is presented below and documented here:

POST /v3/clusters/cluster-1/topics HTTP/1.1
Host: kafkaproxy.example.com
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json

{
  "data": {
    "attributes": {
      "topic_name": "topic-1",
      "partitions_count": 2,
      "replication_factor": 3,
      "configs": [
        {
          "name": "cleanup.policy",
          "value": "compact"
        }
      ]
    }
  }
}

Using curl:

curl -X POST -H "Content-Type: application/vnd.api+json" -H "Accept: application/vnd.api+json" \
          --data '{"data":{"attributes": {"topic_name": "topic-1", "partitions_count": 2, "replication_factor": 1, "configs": [{"name": "cleanup.policy","value": "compact"}]}}}' \
          "http://localhost:8082/v3/clusters/<cluster-id>/topics"

where the cluster-id can be identified using

curl -X GET -H "Accept: application/vnd.api+json" localhost:8082/v3/clusters

Upvotes: 13

Linda
Linda

Reputation: 88

You can create a topic with Azure HdInsight Kafka REST proxy. https://learn.microsoft.com/en-us/rest/api/hdinsight-kafka-rest-proxy/admin

More details here: https://learn.microsoft.com/en-us/azure/hdinsight/kafka/rest-proxy

Upvotes: 0

azary
azary

Reputation: 61

There is no explicit admin API to create a topic in REST proxy yet, but if the Kafka broker property auto.crate.topics.enable is set to true the topic is created when you first post to it, including the case when the post comes from the proxy. More info here How to create topics in apache kafka?

Upvotes: 2

David Griffin
David Griffin

Reputation: 13927

No. The problem is there's no protocol support for creating topics (not yet, at least). See my previous question here, and read the comments.

Upvotes: 3

Related Questions