CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectdebezium
Osama
Osama

Reputation: 476

kafka connector debezium stuck at snapshot of large data

I setup elasticsearch, kibana, mongodb, and kafka on the same linux server for development purposes. The server has 30GB Memory and enough disk space. I'm using a debezium connector and I'm trying to copy a large collection of about 70GB from mongodb to elasticsearch. I have set memory limits for each of elasticsearch, mongodb, and kafka, because sometimes one process will use up the available system memory and prevent the other processes from working.

The debezium connector seemed to be working fine for a few hours as it seemed to be building a snapshot as the used disk space was consistently increasing. However, the disk usage has settled at about 45GB and is not increasing.

The connector and tasks status is RUNNING.

There are no errors or warnings from kafka connectors, which are running in containers.

I tried increasing the memory limits for mongodb and kafka and restarting the services, but no difference was noticed.

I need help troubleshooting this issue.

Upvotes: 1

Views: 190

Answers (0)

Related Questions