CODEWITHSUNDEEP
apache-sparkapache-spark-sql
ZMath_lin
ZMath_lin

Reputation: 553

update query in Spark SQL

I wonder can I use the update query in sparksql just like:

sqlContext.sql("update users set name = '*' where name is null")

I got the error:

org.apache.spark.sql.AnalysisException: 
Unsupported language features in query:update users set name = '*' where name is null

If the sparksql does not support the update query or am i writing the code incorrectly?

Upvotes: 15

Views: 61055

Answers (1)

Pranav Shukla
Pranav Shukla

Reputation: 2226

Spark SQL doesn't support UPDATE statements yet.

Hive has started supporting UPDATE since hive version 0.14. But even with Hive, it supports updates/deletes only on those tables that support transactions, it is mentioned in the hive documentation.

See the answers in databricks forums confirming that UPDATES/DELETES are not supported in Spark SQL as it doesn't support transactions. If we think, supporting random updates is very complex with most of the storage formats in big data. It requires scanning huge files, updating specific records and rewriting potentially TBs of data. It is not normal SQL.

Upvotes: 18

Related Questions