CODEWITHSUNDEEP
google-cloud-dataflowpipelineapache-beam
LaurensVijnck
LaurensVijnck

Reputation: 552

Force update of SideInput on updating Dataflow pipeline

I have a Dataflow pipeline running that fetches a configuration of active tenants (stored in GCS) and feeds it into an ActiveTenantFilter as a sideInput. The configuration is rarely updated, hence why I decided to re-deploy the pipeline, using the --update flag, whenever it is updated.

However, when using the update flag, the file is not fetched again, i.e., the state is maintained. Is it possible to enforce that this PCollectionView is updated whenever the pipeline is re-deployed?

Pipeline Description

Upvotes: 0

Views: 264

Answers (1)

robertwb
robertwb

Reputation: 5104

You are correct, when you --update a pipeline it will process new data but will not re-load old data. It sounds like what you want is slowly updating side inputs which unfortunately has not been implemented yet. You could instead try draining and re-starting your pipeline.

Upvotes: 1

Related Questions