Read more than 1000 rows from Azure Table Storage OData filter query?

Question

How do we read more than 1000 rows from Azure Table Storage? Here is the code I'm using for reading data from Table Storage and this only retrieves 1000 rows:

readData()
{
    var s = @$"https://{storageAccountName}.table.core.windows.net/{tableName}()";          
    var baseurl = $"{s}<sas-token>&$filter=Name%20eq%20'XYZ'";
    var data = GetForOData(baseurl);
    var responseData = data.Data.Replace(".", "_");
    var odata = JsonConvert.DeserializeObject<ODataResponse>(responseData); 
}

GetForOData(string url) {return InvokeForOData<Object>("GET", url, null, null);}

private static HttpResponseData InvokeForOData<T>(string method, string url, Object id, T data)
{
    var Response = new HttpResponseData()
    {
         Code = HttpStatusCode.RequestTimeout, Data = string.Empty, Message = string.Empty
    };
            
    var PostParam = string.Empty;
    if (data != null) { PostParam = data.ToString(); }
    var postData = Encoding.UTF8.GetBytes(PostParam);
    var request = (HttpWebRequest)WebRequest.Create(new Uri(url + (id == null ? "" : '/' + id.ToString())));
     request.Method = method;
     // add headers
     if (postData.Length > 0)
     {
        using (Stream requestStream = request.GetRequestStream())
        { requestStream.Write(postData, 0, postData.Length); }
     }
     using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
     {
        Response.Code = response.StatusCode;
        using (var stream = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
        { Response.Data = stream.ReadToEnd(); }
     }            
     return Response;
}

Where do I check for x-ms-continuation-NextPartitionKey and x-ms-continuation-NextRowKey and use them in the next request?

Update: I was able to find nextPartitionKey and nextRowKey header values. How do I pass these values in the next request?

                using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                {
                    Response.Code = response.StatusCode;
                    var nextPartitionKey = response.Headers["x-ms-continuation-NextPartitionKey"];
                    var nextRowKey = response.Headers["x-ms-continuation-NextRowKey"];
                    using (StreamReader stream = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
                    {
                        Response.Data = stream.ReadToEnd();
                    }
                }

Answer 1

A single call to Azure Table Storage will return a maximum of 1000 entities. If there are more entities matching a query, you will get a continuation token back and you will need to use that to get the next set of entities.

So in your case, in order to read more than 1000 entities you will have to send the request, get the data and check for continuation token in the response (x-ms-continuation-NextPartitionKey and x-ms-continuation-NextRowKey) and use them in the next request.

You can learn more about pagination in Azure Table Storage here: https://learn.microsoft.com/en-us/rest/api/storageservices/query-timeout-and-pagination.

UPDATE

Please see the code below (untested though):

private static HttpResponseData InvokeForOData<T>(string method, string url, Object id, T data)
{
  var Response = new HttpResponseData()
  {
        Code = HttpStatusCode.RequestTimeout, 
        Data = string.Empty, 
        Message = string.Empty,
        NextPartitionKey = string.Empty,
        NextRowKey = string.Empty
  };
            
  var PostParam = string.Empty;
  if (data != null) { PostParam = data.ToString(); }
  var postData = Encoding.UTF8.GetBytes(PostParam);
  var request = (HttpWebRequest)WebRequest.Create(new Uri(url + (id == null ? "" : '/' + id.ToString())));
    request.Method = method;
    // add headers
    if (postData.Length > 0)
    {
      using (Stream requestStream = request.GetRequestStream())
      { requestStream.Write(postData, 0, postData.Length); }
    }
  using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
  {
      Response.Code = response.StatusCode;
      var nextPartitionKey = response.Headers["x-ms-continuation-NextPartitionKey"];
      var nextRowKey = response.Headers["x-ms-continuation-NextRowKey"];
      using (StreamReader stream = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
      {
          Response.Data = stream.ReadToEnd();
          Response.NextPartitionKey = nextPartitionKey;
          Response.NextRowKey = nextRowKey;
      }
  }     
  return Response;
}

Basically what I have done is added two properties (NextPartitionKey and NextRowKey) in your HttpResponseData object and populate that with the nextPartitionKey and nextRowKey respectively you get from response header.

What you have to do in your code (where you're processing the response) is check if either of these two values are not null or empty. A non null/empty value would indicate that more entities are present.

If that's the case, then what you have to do is modify the URL by appending NextPartitionKey and NextRowKey values as query string parameters (please see the link above for details) and send the request again. You will need to do this till the time you get both of these values as either null or empty strings.