CODEWITHSUNDEEP
google-cloud-dataflowapache-beam
Bluecrow
Bluecrow

Reputation: 609

Determine if a pcollection is empty or not

How to check if a pcollection is empty or not before writing out to a text file in apache beam(2.1.0)?

What i'm trying to do here is to break a file into pcollections of specified number given as a parameter to the pipeline via ValueProvider. As this ValueProvider is not available at pipeline construction time, i declare a decent no 26(total no of alphabets and this is the max no which a user can input) to make it available for .withOuputTags(). So I get 26 tuple tags from which i have to retrieve pcollections before writing to text files. So here, only few number of tags as inputted by user will get populated and rest all are empty. Hence want to ignore empty pcollections returned by some of the tags before i apply TextIO.write().

Upvotes: 0

Views: 1652

Answers (1)

jkff
jkff

Reputation: 17913

It seems like actually you want to write a collection into multiple sets of files, where some sets may be empty. The proper way to do this is using the DynamicDestinations API - see TextIO.write().to(DynamicDestinations) which will be available in Beam 2.2.0 which should be cut within the next couple of weeks. Meanwhile if you'd like to use it, you can build a snapshot of Beam at HEAD yourself.

Upvotes: 1

Related Questions