CODEWITHSUNDEEP
apache-sparkdatabricksspark-structured-streamingdelta-lake
Somasundaram Sekar
Somasundaram Sekar

Reputation: 5524

Delta table versioning while writing from a Spark structured streaming job

Will writing to a Delta table from a Spark structured streaming job create a version for every micro batch of data written?

Upvotes: 3

Views: 2568

Answers (1)

Ged
Ged

Reputation: 18108

From the docs:

As you write into a Delta table or directory, every operation is automatically versioned.

So, yes you are correct.

Getting the data:

  • Using a timestamp

  • Using version number

Reference: https://databricks.com/blog/2019/02/04/introducing-delta-time-travel-for-large-scale-data-lakes.html

Upvotes: 3

Related Questions