CODEWITHSUNDEEP
apache-sparkpysparkdatabricksdelta-lakeshallow-copy
S.Khan
S.Khan

Reputation: 71

Copying rows from multiple delta tables into one via Spark

I have multiple delta lake tables storing images data. Now I want to take specific rows via filter from those tables and put them in another delta table. I do not want to copy the original data just the reference or shallow copy. I am using pyapark and databricks. Can someone please help me find the correct approach for this?

Upvotes: 1

Views: 493

Answers (1)

Alex Ott
Alex Ott

Reputation: 87224

What you actually need is a view over the original table. Use CREATE VIEW to create it with necessary filter expression, like this:

CREATE VIEW <name> AS
SELECT * from <source_table> WHERE <your filter condition>

Then this view could be queried like a normal table, but data will be filtered according to your condition.

Upvotes: 0

Related Questions