CODEWITHSUNDEEP
google-cloud-dataflow
jhpAZ
jhpAZ

Reputation: 1

How do I specify multiple experiments in a flex template run?

I am using Dataflow flex templates and am trying to launch as a job with GPU. I am following docs here to build my template from base nvidia image: https://cloud.google.com/dataflow/docs/guides/using-gpus

I want to run the template with a GPU attached. This requires experiments:

I believe these need to be specified separately rather than as a list. Or at least I haven't found a way to do that and the docs specify two --experiment arguments. From looking at docs I also believe that I need to specify the experiments as part of the --parameters argument as there is no --experiments argument for running flex templates.

I have tried the following:

In gcloud command line:

This caused an error:

   ERROR: (gcloud.dataflow.flex-template.run) INVALID_ARGUMENT: The template parameters are invalid. Details:
experiments: Runtime parameter experiments should not be specified in both parameters field and environment field.

I can get each experiment to work separately but cannot get them to both work. Is there a simple fix for this? I haven't been able to find anything in the documentation nor figured it out myself.

Please let me know any additional information you need me to provide.

Upvotes: 0

Views: 1922

Answers (1)

ningk
ningk

Reputation: 1383

You can pass both experiments in additional-experiments:

--additional-experiments=worker_accelerator=type:nvidia-tesla-t4;count:1;install-nvidia-driver,use_runner_v2

--additional-experiments=worker_accelerator=type:nvidia-tesla-t4;count:1;install-nvidia-driver \
--additional-experiments=use_runner_v2

should both work.

Upvotes: 1

Related Questions