Managing large number of application settings in Azure Functions

Question

I currently have an application that processes telemetry events from Azure IOT Hub. The processing of the events are done by triggering Azure Functions from the IOT Hub.

Several processing rules /settings (currently about 6) are set up per device. Settings are essentially just a set of simple (string/bool) properties. Seeing that there are several hundred devices, the qty of these settings explode over time. On-boarding new devices also creates a requirement for provisioning the settings and defaults.

When the Azure Function starts up to process events from a device, it needs to load the settings for that device and then process accordingly.

What is the best way to manage these settings? The primary focus is on efficient loading of the settings when the function starts up. Secondarily, the easy update of settings.

As far as I can think there are the following options:

• Store individual settings in Azure Tables. I have implemented this but I am concerned that it will not scale well when hundreds of devices access the Azure Table. I have used PartitionKey|RowKey as DeviceId|SettingName. This will create a partition per device

  • Store all settings for a device as JSON in Azure tables. This will require parsing JSON to obtain the individual settings

  • Storing individual settings in ApplicationSettings is not really doable I think due to number of settings

  • Storing all settings for a device I think is also not doable due to the same reason as above.

Answer 1

It sounds like these settings may change, which means you either need to:

1. Read them every time a function starts.
2. Read them once, and store them in memory. Then you'd need an expiration policy to refresh them if they've changed (unless you have some way to be notified of changes). Perhaps something like MemoryCache.
3. Send the device settings with your messages -- is this a possibiliy, or are those managed separately?

As for where they're stored -- using Azure Tables or even Azure Blobs would work. You could store JSON settings in a blob named by the device id and retrieve those settings very quickly. Storing them in something like CosmosDB (which is a JSON document store already) would also work. All of these should scale for you, especially if you're doing some kind of in-memory caching so you don't need to read settings every invocation.