CODEWITHSUNDEEP
tensorflowapache-sparkgoogle-cloud-dataproc
Amit Ghosh
Amit Ghosh

Reputation: 63

Error: org.apache.spark.SparkException: No executor resource configs were not specified for the following task configs: gpu

I am trying to use tensorflow multiworker strategy on GCP dataproc spark cluster with only CPU using below link: https://cloud.google.com/blog/products/data-analytics/faster-machine-learning-dataproc-new-initialization-action

but after spinning up the cluster whenever I am trying create a spark session i am getting Error: org.apache.spark.SparkException: No executor resource configs were not specified for the following task configs: gpu

spark session code:

from pyspark.sql import SparkSession
from spark_tensorflow_distributor import MirroredStrategyRunner

spark = (SparkSession.builder.appName("tf")
  .config("spark.dynamicAllocation.enabled", "false")
  .config("spark.executor.resource.gpu.amount", "0")
  .config("spark.executor.cores", "4")
  .config("spark.task.cpus", "4")
  .config("spark.task.resource.gpu.amount", "0")
  .config("spark.driver.resource.gpu.amount", "0")
  .config("spark.executor.resource.gpu.discoveryScript", "/usr/lib/spark/scripts/gpu/getGpusResources.sh")
  .config("spark.driver.resource.gpu.discoveryScript", "/usr/lib/spark/scripts/gpu/getGpusResources.sh")
  .getOrCreate())

how do I resolve this?

Upvotes: 2

Views: 1384

Answers (1)

Brad Miro
Brad Miro

Reputation: 251

If you're not intending to use GPUs for your training, you don't need to specify any configuration pertaining to GPUs.

Try with just the following:

  spark = (SparkSession.builder.appName("tf")
    .config("spark.dynamicAllocation.enabled", "false")
    .config("spark.executor.cores", "4")
    .config("spark.task.cpus", "4")
    .getOrCreate())

Upvotes: 1

Related Questions