CODEWITHSUNDEEP
azure-iot-hubazure-iot-sdk
Yoseph
Yoseph

Reputation: 2416

How to implement paging in Azure Iot device twin queries

The azure documentation at https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-query-language says that “Azure IoT SDKs support paging of large results” but I could not find any sample or reference on how to do it.

Does anyone have an idea?

Upvotes: 2

Views: 1801

Answers (3)

Itay Grudev
Itay Grudev

Reputation: 7434

After some reverse-engineering the IoTHub client code I found the following approach to work.

It uses the continuationToken from the query object and passes it to either next or nextAsTwin calls.

After fetching the last page query.continuationToken is set to undefined so you can use that as your termination condition.

const query = registry.createQuery("SELECT * FROM ...", pageSize: 10000)
do {
  /* eslint-disable no-await-in-loop */
  const {result} = await query.next(query.continuationToken);

  your.code(here)

  /* eslint-enable no-await-in-loop */
} while( query.continuationToken !== undefined )

P.S. If you need to fetch all the pages you might as well set pageSize to maximum to reduce the network overhead and improve performance.

Upvotes: 0

Yoseph
Yoseph

Reputation: 2416

Using Azure IoT device SDK for Node.js

var Registry = require('azure-iothub').Registry;
var connectionString = '{iothub connection string}';
var registry = Registry.fromConnectionString(connectionString);
var pageSize = 10;
var query = registry.createQuery("SELECT * FROM devices", pageSize);

To get the first page:

query.next(function (err, devices, response) {
    if (err) {
        console.error('Failed to query devices: ' + err.message);
    } else {
        var continuationToken = response.headers["x-ms-continuation"]; // Example: "c2tpcD0wJnRvdGFsPTEwJmxhc3Q9ZGV2aWNlMTA="
        var pageToken = new Buffer(continuationToken, 'base64').toString('ascii'); // Example: "skip=0&total=10&last=device10"
        //Optionally, you may persist the token and use it for the next pages
    }
});

To get the next page,

query.next(continuationToken , function (err, devices, response) {…} //previous token

To get the fourth page

var pageNumber = 3; // zero based
var pageToken = "skip=" + pageNumber * pageSize + "&total=" + pageSize; // "skip=30&total=10"
var continuationToken = new Buffer(pageToken).toString('base64'); //"c2tpcD0zMCZ0b3RhbD0xMA=="
query.next(continuationToken, function (err, devices, response) {
    if (err) {
        console.error('Failed to query devices: ' + err.message);
    } else {
        //…
    }
});

Using Azure IoT service SDK for .NET

Install Microsoft.Azure.Devices nuget package

    string connectionString = "{iot hub connection string}";
    int pageSize = 10;
    var registryManager = RegistryManager.CreateFromConnectionString(connectionString);
    var query = registryManager.CreateQuery("SELECT * FROM devices", pageSize);

    Console.WriteLine("First page");
    var firstPage = query.GetNextAsTwinAsync();
    var response = (QueryResponse<Microsoft.Azure.Devices.Shared.Twin>)firstPage.Result;
    var continuationToken1 = response.ContinuationToken;
    response.ToList().ForEach(d => Console.WriteLine(d.DeviceId));

    Console.WriteLine("Next page");
    var nextPage = query.GetNextAsTwinAsync(new QueryOptions() { ContinuationToken = continuationToken1 });
    nextPage.Result.ToList().ForEach(d => Console.WriteLine(d.DeviceId));

    Console.WriteLine("Fourth page");
    var pageNumber = 3; // zero based
    var pageToken = "skip=" + pageNumber * pageSize + "&total=" + pageSize; // "skip=30&total=10"
    var continuationToken3 = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(pageToken)); //"c2tpcD0zMCZ0b3RhbD0xMA=="
    var fourthPage = query.GetNextAsTwinAsync(new QueryOptions() { ContinuationToken = continuationToken3 });
    fourthPage.Result.ToList().ForEach(d => Console.WriteLine(d.DeviceId));

Note: I don't know why but I got "missing API 2!" error when I use .NET Core.

Upvotes: 3

Roman Kiss
Roman Kiss

Reputation: 8255

It's based on the REST API POST calls and headers such as x-ms-max-item-count and X-Ms-Continuation , see the following screen snippets:

query-firstpage

query-nextpage

query-lastpage

as you can see the above last picture doesn't return a continuation header, therefore this page is last one.

Upvotes: 5

Related Questions