CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectconfluent-platform
NoName
NoName

Reputation: 1759

Kafka: Does Confluent’s HDFS connector support Snappy compression?

I don't see any configurations for compression on the HDFS connector docs https://docs.confluent.io/current/connect/connect-hdfs/docs/configuration_options.html.

Upvotes: 0

Views: 625

Answers (1)

Konstantine Karantasis
Konstantine Karantasis

Reputation: 1993

Snappy compression was recently added to the HDFS Connector for Avro. To enable it you'll need to set the property avro.codec to snappy.

With Parquet it's been available since the beginning and it is the codec used when exporting parquet files.

Here's the PR that added it for Avro:

https://github.com/confluentinc/kafka-connect-hdfs/pull/255

This config property will become available in the forthcoming 4.1.0 release of the connector. That's also when the link to the docs will be updated to reflect this addition.

Upvotes: 1

Related Questions