Azure Search Options (Have I understood Azure Search prcicng)

Question

We have 100 million rows in table storage, each row has about 4 items of metadata, we would like to search by the metadata, looking at the pricing it going to be very expensive.

The Basic option supports 1M documents, is that the same as 1 million rows with only 4 items of metadata? Or would we simply use up the storage which is 2GB which would be optomised so 2GB of metadata may not = 2GB of stroage.

And then the bigger size

We are also looking at Document DB and the standard pattern with table storage to allow search on other metadata.

Answer 1

Basic has a hard limit of 1M document, and you cannot add more partitions to increase that so you have to go for one of the Standard tiers, S1, S2 or S3 if you want to index all 100M entries. Each individual entry (row in your database) counts as a document. The maximum size of the documents you index is 16 MB, but it may be lower depending on how you update the index (https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity#document-size-limits).

The number of documents you need to store affects which tier you need, but also the maximum storage size and which throughput you want. You could do a quick estimate of how big your storage size needs to be, e.g. if your 4 metadata points are all string and each string is on average 30 chars of UTF-8 then you need a total of (100M x 4 x 30 b) ~ 11 GB so storage size will likely not be something you need to select more partitions for (both S1 and S2 can fit that within a single partition).

To fit 100M documents you could go with 7 S1 partitions (15M x 7 = 105M) at £1,304.21/mo, or 2 S2 partition (100M x 2) at £1,490.52/mo. The S2s will likely give you better throughput and it will give you more indexes to work with (even if you don't need them at the moment (since you only have 4 metadata points).

As noted before, the full capability of a search engine might be a lot more than what you need right now, but if it is a strategic decision to start working with it then at least you know why you are paying for it.

Answer 2

If your search scenarios don't require full text search (e.g. type ahead, suggestions, stemming and inflected forms of words in over 50 languages, facets, custom scoring), but instead need just numerical / datetime / geospatial comparisons and simple string matching, then DocumentDB would be a good choice. DocumentDB is also easy to efficiently integrate with Azure Search if / when you need full text search capabilities.

In terms of Azure Search pricing, look at S1 and S2 tiers. For example, you can store 100 million docs with 2 S2 partitions. The storage and document limits shown on pricing tiles are per partition. Each service can have up to 12 partitions, except Basic that only has 1.