CODEWITHSUNDEEP
apache-sparkstreamingspark-streamingazure-databricks
play_something_good
play_something_good

Reputation: 141

streaming in a Medallion Architecture with a Star Schema?

What are the best practices for implementing non-stop streaming in a Medallion Architecture with a Star Schema?

Use Case:

We have operational data and need to enable near real-time reporting in Power BI, with a maximum latency of 3 minutes. No Delta Live tables.

Key Questions:

How should we curate dimensions and facts when transitioning data from Silver to Gold using Structured Streaming? What are the best practices for managing slowly changing dimensions (SCDs) in a streaming pipeline? Could you provide examples or proven approaches for fact-dimension joins in a streaming context?

Upvotes: 0

Views: 54

Answers (0)

Related Questions