CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
Kalyan
Kalyan

Reputation: 1940

How to run update queries on spark-sql

I am newbie in spark.Is there anyway that i could operate Update command in spark-SQL. I have already created a hive table where i have manipulated some SQL syntax queries like insert, select ,delete but could not able to run update command. I have searched on spark documentation and https://docs.datastax.com/en/datastax_enterprise/4.6/datastax_enterprise/spark/sparkSqlSupportedSyntax.html site for this. but i did not find UPDATE Syntax in their recent list of Spark SQL. Kindly help me.I am using spark 2.0

Kalyan

Upvotes: 3

Views: 4701

Answers (5)

Jaime Caffarel
Jaime Caffarel

Reputation: 2469

The general answer is that you can't. However, if the data is stored in Hive using ORC, you could do something like this to update the Table using Spark SQL.

Upvotes: 2

Dyuti
Dyuti

Reputation: 184

HDFS is a write once file system and ORC is a write-once file format, so edits were implemented using base files and delta files where insert, update, and delete operations are recorded.

Upvotes: 0

anshul_cached
anshul_cached

Reputation: 762

DataFrame records don't have any setter methods because DataFrames are based on RDD which are immutable collections, which means you cannot change their state. Thats is why spark sql does not support updates.

Upvotes: 3

Sandeep Purohit
Sandeep Purohit

Reputation: 3702

No you cant run update query in spark-hive, its not support transactional queries.

Upvotes: 0

maxymoo
maxymoo

Reputation: 36555

No you can't do update queries, you'll have to work out how to do it as a SELECT.

Upvotes: 1

Related Questions