CODEWITHSUNDEEP
amazon-s3apache-kafkaapache-kafka-connectaws-msk
Don Woodward
Don Woodward

Reputation: 131

Kafka Connect SpooldirCsv connector

I am trying to use the Confluent's SpoolDirCSVSource connector to read files from a directory and send to MSK. Works fine on my local. But now I need to source it from S3 bucket . Is there no way I can use this connector to do this ? Or is there some other connector which does this ? The input.path parameter works only with local directories I think. Any pointer in right direction to the correct connector or modifying the SpoolDirCSV connector would be appreciated.

I know this question is old , something similar. But I am curious to know if this functionality is still absent (I can be wrong).

How to use Kafka Connect to source .csv files from S3 bucket?

This is the exact error when the connector is deployed to cloud

There is an issue with the connector Code: InvalidInput.InvalidConnectorConfiguration Message: The connector configuration is invalid. Message: Connector configuration is invalid and contains the following 2 error(s): Invalid value File 's3:/mytestbucketak/input' is not an absolute path. for configuration input.path Invalid value File 's3:/mytestbucketak/error' is not an absolute path. for configuration error.path

Upvotes: 0

Views: 143

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

That connector can only read from local filesystem, not S3.

Confluent has a specific S3 Source Connector, or as linked, FilePulse Connector also exists.

Upvotes: 1

Related Questions