CODEWITHSUNDEEP
mongodbapache-kafkaapache-kafka-connect
Ricardo Moreira
Ricardo Moreira

Reputation: 41

mongodb kafka source connector - pipeline not working as expected

I'm trying to push to Kafka only documents from MongoDB that match specific criteria. When I don't add any configuration for the "pipeline" property, I get all the documents to be pushed to Kafka. When I add a configuration to filter for matching criteria, I get none. Here is my connector configuration:

name=MongoDBSourceConnector
tasks.max=1
connector.class=com.mongodb.kafka.connect.MongoSourceConnector
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
connection.uri=mongodb://myconnectionstring
database=mydb
collection=mycollection
publish.full.document.only=true
pipeline=[ { $match: { title: "mytitle"} } ]
copy.existing.pipeline=[ { $match: { title: "mytitle"} } ]
copy.existing=true

I'm following this documentation: https://docs.mongodb.com/kafka-connector/current/kafka-source/ I get no error on Kafka Connector logs. Any idea on what am I doing wrong here?

Upvotes: 2

Views: 831

Answers (1)

Ricardo Moreira
Ricardo Moreira

Reputation: 41

The problem is with the "". MongoDb operators should be with no quotes and fields should have it:

pipeline=[ { $match: { "title": "mytitle"} } ]

copy.existing.pipeline=[ { $match: { "title": "mytitle"} } ]

Upvotes: 2

Related Questions