CODEWITHSUNDEEP
pythongoogle-cloud-platformapache-kafka
윤태일
윤태일

Reputation: 557

How can I connect to the kafka server from an external server?

I have a kafka cluster of 3 instances on Google Cloud.

I want to configure consumer via python file on another external server.

Here is the python code:

from kafka import KafkaConsumer
from json import loads


if __name__ == "__main__":

    topic_name = "test"
    consumer = KafkaConsumer(
        topic_name,
        bootstrap_servers=[
            "externalIP:9092",
            "externalIP:9092",
            "externalIP:9092",
        ],
        auto_offset_reset="latest",
        enable_auto_commit=True,
        group_id="test-consumer-group",
        value_deserializer=lambda x: loads(x.decode("utf-8")),
        consumer_timeout_ms=1000,
    )
    print(consumer.topics())

    print("[begin] get consumer list")

    for message in consumer:
        print("test")
        print(
            "Topic: %s, Partition: %d, Offset: %d, Key: %s, Value: %s"
            % (
                message.topic,
                message.partition,
                message.offset,
                message.key,
                message.value,
            )
        )
        print("[end] get consumer list")

The config/server.properties file configured on the GCP instance looks like this:

broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://kafka-1:9092

I changed the config file according to all cases that came up in the search, but only the topic was printed in the Python code and data could not be received.

(Data collection is confirmed in real time within the same server as the producer)

I've tried disconnecting internal and external, and I've tried everything, but I can't get it. Please help me

Upvotes: 0

Views: 1513

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191844

You need to advertise an externally resolvable address.

Please refer - https://www.confluent.io/blog/kafka-listeners-explained/

If you're interested in using GKE Kubernetes, you could deploy Strimzi operator, and it can be used to configure externally resolvable addresses, among other features, for you.

Upvotes: 1

Related Questions