CODEWITHSUNDEEP
apache-sparkgoogle-cloud-platformpysparkgoogle-bigquerydataproc
Nicolas Soria
Nicolas Soria

Reputation: 51

Error While Writing into a BigQuery table from Dataproc - Spark

I am trying to insert data into a Big Query table from Dataproc - Spark. I'm using the following code:

df.write \
  .format("bigquery") \
  .option("temporaryGcsBucket","bucket/temp") \
  .save("project.datasource.table1")

Error:

pyspark.sql.utils.IllegalArgumentException: 'SaveMode is set to ErrorIfExists and Table project.datasource.table1 already exists. Did you want to add data to the table by setting the SaveMode to Append? Example: df.write.format.options.mode(SaveMode.Append).save()'

I inserted ".mode(SaveMode.Append)", but It didn't work.

Upvotes: 2

Views: 1962

Answers (1)

Pav3k
Pav3k

Reputation: 909

That will do:

df.write \
  .format("bigquery") \
  .option("temporaryGcsBucket","bucket/temp") \
  .mode("append") \
  .save("gcp-bankier.sof.table1")

Upvotes: 1

Related Questions