CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
bismi
bismi

Reputation: 165

Kafka FileStreamSinkConnector recreated the file to write in after it was deleted by me

OK, this is an unusual one.

I made a File stream sink connector as follows:

curl -X POST http://cpnode.local.lan:8083/connectors -H "Content-Type: application/json" --data '{
  "name":"file-sink-connector-002",
  "config": {
    "tasks.max":"1",
    "batch.size":"1000",
    "batch.max.rows":"1000",
    "poll.interval.ms":"500",
    "connector.class":"org.apache.kafka.connect.file.FileStreamSinkConnector",
    "file":"/kafka/names.txt",    
    "table.name.format":"tb_sensordata",
    "topics":"names",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "key.converter.schemas.enable":"false",
    "value.converter": "org.apache.kafka.connect.storage.StringConverter",
    "key.converter.schemas.enable":"false"
  }
}'

While the connector was running, I deleted the file names.txt

After an hour or so.....It recreated the file.....

I started console producer and produced some data in the topic. The sink connector wrote the data in the file.

Can anyone explain this behavior....plz.

Upvotes: 0

Views: 884

Answers (1)

Michael Heil
Michael Heil

Reputation: 18495

According to this pull request (MINOR: Append or create file in FileStreamSinkTask #5406) the file mentioned in a FileStreamSinkConnector will get created by kafka-connect if it does not exists.

Upvotes: 1

Related Questions