CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectconfluent-platform
user12431003
user12431003

Reputation: 31

With Kafka JDBC source connector getting only 1000 records/sec. How to improve the record pulling rate

I am using kafka connect with JDBC source connector. The connector is working fine but i able to get only 1000 messages/sec. to the topic from Oracle DB. I tried most of the configuration settings but no luck. i tried in both standalone and distributed modes. Pls. help on this. Below is my JDBC Source connector configuration:

curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{"name": "ORA_SRC_DEVDB",
"config": {                "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",                 
"connection.url": "jdbc:oracle:thin:@xxxxxxx/DBDEV",                 
"connection.user": "xxxxxx",                 
"connection.password": "xxxxxx",                 
"query": "select * from A.LOG_AUDIT",               
"topic.prefix": "Topic_POC", 
"tasks.max": "1",  
"poll.interval.ms": "5000", 
"batch.max.rows": "1000", 
"table.poll.interval.ms": "60000",  
"mode": "timestamp",    
"timestamp.column.name": "MODIFIED_DATEnTIME"                 }        
 }'

And also the destination Topic "Topic_POC" created with 3 partitions & 3 replicas.

Upvotes: 3

Views: 2746

Answers (1)

Nitin
Nitin

Reputation: 3842

poll.interval.ms: Frequency in ms to poll for new data in each table(default 5000)

batch.max.rows: Maximum number of rows to include in a single batch(default 100)

In your case every 5 seconds you are polling max 1000 record from DB. Trying to decrease poll.interval.ms and increase batch.max.rows could improve fetch rate.

Not only that below factors also impact on your fetch rate

  1. Rate of incoming data into the Database that also depends
  2. I/O rate from DB to JDBC connector to Kafka
  3. DB table performance if you have a proper index on the time column.
  4. After all its uses JDBC to fetch data from the database so implies all that you face on a single JDBC application

As per my experience JDBC connector is pretty fas

Upvotes: 1

Related Questions