CODEWITHSUNDEEP
apache-flinkflink-streaming
YuFeng Shen
YuFeng Shen

Reputation: 1565

Why we need StreamingFileSink when there is exists BucketingSink?

I found BucketingSink can do everything the StreamingFileSink can do, such as write events to local files, NAS or HDFS files(Original I thought the BucketingSink just can write events to HDFS and cannot write to NAS or local files), however some functions StreamingFileSink doesn't provides which BucketingSink provides , such as the StreamingFileSink cannot set the Prefix and Suffix of the file, however BucketingSink can do, so Why we need StreamingFileSink when there is exists BucketingSink?

Upvotes: 0

Views: 423

Answers (1)

Fabian Hueske
Fabian Hueske

Reputation: 18987

StreamingFileSink is a new sink that will eventually replace BucketingSink. BucketingSink has some major limitation (consistent output to S3, support for blocking formats such as Parquet and ORC) that cannot be resolved without a redesign. StreamingFileSink implements this new design.

However, as you've noticed in the current state (Flink 1.6), StreamingFileSink does not support all features of BucketingSink yet.

Upvotes: 1

Related Questions