CODEWITHSUNDEEP
powerbiquery-optimizationgoogle-cloud-dataflow
Bryan Rock
Bryan Rock

Reputation: 632

Power BI Dataflow with Incremental Refresh works, but is slow to load

I need some advice for using Power BI dataflows with incremental refresh.

I have 12 fact tables (general ledger entries for each state in the northeast.) I need to append them together into a master table. When combined, it's about 50 million rows. Right now I'm using a dataflow to achieve this. I use incremental refresh on the individual tables AND on the master table. This seems to work and only takes about 10 minutes to refresh each night.

The problem is that loading this data into my dataset takes at least an hour. Just wondering what best practices are here. Should I be using incremental refresh on the dataset as well? Or should I ditch dataflows and just do everything in the dataset? The only info I was able to find online is a few years old.

(Also, the workspace is on a premium capacity.)

Upvotes: 0

Views: 981

Answers (1)

David Browne - Microsoft
David Browne - Microsoft

Reputation: 88996

Should I be using incremental refresh on the dataset as well?

If you have a long-running refresh and you can easily implement incremental refresh, you should. In addition to enabling you to load only changed data, it also speeds up full refreshes by refreshing partitions in parallel.

Upvotes: 1

Related Questions