CODEWITHSUNDEEP
pysparkdatabricksdelta-live-tables
Luke88
Luke88

Reputation: 55

Databricks Delta Live Table - How To Simply Append A Batch Source To a DLT Table?

Using Python and all the relevant DLT properties within Databricks, does anyone know how to simple append to a DLT table from a batch source?

In PySpark you can just use df.write.format("delta").mode("append") but since dlt requires you to return a Spark dataframe within its decorator, we can't use the Dataframe Writer API.

Thanks in advance.

Upvotes: 4

Views: 4637

Answers (1)

Alex Ott
Alex Ott

Reputation: 87174

Delta Live Tables has a notion of a streaming live table that is append-only by default. You can define your pipeline as triggered, that will be equivalent of the the Trigger.Once. Something like that:

@dlt.table
def append_only():
  return spark.readStream.format("xyz").load()

Here we use readStream just to make sure that when we run the pipeline again we won't append the same content again & again.

Upvotes: 1

Related Questions