CODEWITHSUNDEEP
azureazure-blob-storage
dragansr
dragansr

Reputation: 480

Azure Append Blobs Slow Read Performance

I have observed that reading from Azure Append Blob is very slow when a blob is appended to a few thousand times or more. Writing/appending is fast, but reading a typical log blob with a few thousand appends each of size a few KB, for total size of a few MB takes more than a minute! Reading standard blog or page blob of similar size takes milliseconds. Is there some way to speed up reading from append blob, i.e. by flattening internal structure?

So far it looks like that it may be best to periodically "archive" / convert append blobs to block blobs, and then process them. Any suggestion?

If not, what would be recommended alternative storage for logs? Azure table can be used, but it would take many more reads, even when bulk operations are used.

Upvotes: 6

Views: 3105

Answers (3)

Mårten Wikström
Mårten Wikström

Reputation: 11344

This is because many small appends will cause the blob to become heavily fragmented, trading fast writes for slow reads.

Currently your best option is, like you wrote, to periodically copy the append blob into a block blob and read from that instead.

However, the Azure storage team have plans (since june 2018) to add a de-fragmentation feature. With that in place we may finally be able to get fast reads for small appends too.

Upvotes: 3

dragansr
dragansr

Reputation: 480

I did switch to Azure Tables and read performance there is reasonable, about 1 second for 1.5K items, read in bulk mode. Still, reading a block or page blob with same content is in milliseconds, much faster. It would be quite effective if there was a way to append to a page blob. That can be done manually, so maybe Append (or some other type of) Blob can automate that in some future version. Append Blobs are quite complex inside Deep dive in Append Blob, that is likely reason for slow read.

Upvotes: 1

Jambor - MSFT
Jambor - MSFT

Reputation: 3293

Please pay attention that append Blobs are made up of blocks. A single append blob could contain up to 50,000 blocks of up to 4MB each. if you append the new content to the existed file, it will caused the slow performance of the file. We could find out the commit number like the following screenshot: enter image description here

It is not a good idea to append new content to the file existed in Append blob. Append blob should be used when we need to append content to a blob and we don't care about the order. If you want to use append to log some application information, we can write the new log information to new file and save to Azure append blob. Azure table is also a good service for us to log application information. If we want to find out the log information, we can use Microsoft Azure Explorer.

Upvotes: -1

Related Questions