CODEWITHSUNDEEP
kubeflowkubeflow-pipelines
Jet Basrawi
Jet Basrawi

Reputation: 3245

Aggregate results when using Kubeflow Pipelines kfp.ParallelFor

What is a good pattern for aggregating the results from Kubeflow Pipleine kfp.ParallelFor?

Upvotes: 6

Views: 1285

Answers (2)

user12154148
user12154148

Reputation: 21

Not exactly what you asked for, but our workaround was to write the results of the parallelfor tasks into S3 and simply collect them afterwards in a postprocessing task.

with dsl.ParallelFor(preprocessing_task.output) as plant_item:
                predict_plant='{}'.format(plant_item)
                forecasting_task = forecasting_op(predict_plant, ....).after(preprocessing_task)
postprocessing_task = postprocessing_op(...).after(forecasting_task)

Upvotes: 2

Peter
Peter

Reputation: 141

At the moment this might not be supported:

Support inputs with multiple arguments #1933

Upvotes: 0

Related Questions