CODEWITHSUNDEEP
pythonapache-kafkaminikube
colo
colo

Reputation: 121

How to reach kafka with a producer situated in minikube

So, i'm out of ideas. i would reach a kafka cluster in my localpc with a producer (written in python with kafka-python library) situated on minikube.

the producer code is:

byte_log = str.encode(f"many stuff")
try:
    producer = KafkaProducer(bootstrap_servers=['local-ip:9092'])  
    future = producer.send('flask.logs', byte_log)
    record_metadata = future.get(timeout=10)
    print(f"record_metadata.topic {record_metadata.topic}")
    print(f"record_metadata.partition {record_metadata.partition} ")
    print(f"record_metadata.offset {record_metadata.offset}")

except Exception as e:
    print("[KAFKA-P] bad post")
    raise e

i have try another way to create the producer and send messages :

producer = KafkaProducer(bootstrap_servers=['0.0.0.0:9092'], value_serializer=lambda v: json.dumps(v).encode('utf-8'))

and send with future = producer.send('flask.logs', json.dumps(log))

and for both codes the error is the same :

kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Timeout after waiting for 10 secs.

furthermore the container where the producer has mounted have the same time zone, then there were no problem with timestamp (maybe).

For reach kafka i created an Endpoint and a service for map localhost port 9092 with the port 9092 of the pod the service deployment is (note there is no selector label ): 

 kind: Service
 apiVersion: v1
 metadata:
 name: local-ip 
 spec:
 ports:
 - protocol: TCP
   port: 9092 
   targetPort: 9092 
 type: ClusterIP

and the Endpoint is :

kind: Endpoints
apiVersion: v1
metadata:
 name: local-ip
subsets:
  - addresses:
   - ip: 192.168.99.101  
ports:
 - port: 9092

could be a problem of Kube-DNS ? if yes, how can i locate it ??

Upvotes: 0

Views: 263

Answers (2)

OneCricketeer
OneCricketeer

Reputation: 191844

If Kafka is running on the host, creating k8s resources for it would not do any good.

You need to provide the host IP as the bootstrap servers and configure Kafka server.properties to list that address as well via advertised.listeners; and 0.0.0.0 is not a valid routable IP

Upvotes: 0

mario
mario

Reputation: 11128

As far as I understood you're using Virtualbox as your hypervisor. I did a quick test and it looks like address 192.168.99.101 is available only from the VM. Instead you should use here IP address of your host. It should be also accessible from your VirtualBox VM as it has by default two network adapters: NAT and host-only. You can additionally try to ping it from VM to make sure it is actually available.

Upvotes: 1

Related Questions