CODEWITHSUNDEEP
azure-data-lakeazure-analysis-services
Brandon
Brandon

Reputation: 183

Create single Azure Analysis Services table from many blobs in Data Lake Store

I'm new to analysis services and data lake, working on a POC. I've used data factory to pull in some TSV data from blob storage, which is logically organized as small "partition" blobs (thousands of blobs). I have a root folder that can be thought of as containing the whole table, containing subfolders that logically represent partitioning by, say, customer - these contain subfolders that loggically represent partitioning the customer's data by, say, date. I want to model this whole folder/blob structure as one table in Analysis Services, but can't seem to figure out how. I have seen the blog posts and examples that create a single AAS table from a single ADLS file, but information on other data file layouts seems sparse. Is my approach to this wrong, or am I just missing something obvious?

Upvotes: 1

Views: 1022

Answers (1)

GregGalloway
GregGalloway

Reputation: 11625

This blog post provides instructions on appending multiple blobs into a single table.

Then the part 3 blog post describes creating some Analysis Services partitions to improve processing performance.

Finally this blog post describes connecting to Azure Data Lake Store (as opposed to Azure Blob Storage in the prior posts).

I would use those approaches to create say 20-200 partitions (not thousands) in Azure Analysis Services. Partitions should generally be at least 8 million rows to get optimal compression and performance. I assume that will require appending several blobs together in order to achieve that size.

Upvotes: 1

Related Questions