CODEWITHSUNDEEP
c#google-cloud-storage
S_Luis
S_Luis

Reputation: 521

Why the NextPageToken in Google's StorageClient is always null?

I'm trying to list all items under a path for a determined bucket using Google Cloud Storage APIs. Under this path, there are more than 1000 items - which is the maximum number of items that ListObjects / ListObjectsAsync return.

In order to be able to repeat the call and get the next 1000 items, I need the NextPageToken from the previous response, that's how result pagination gets done! It's easy and reasonable.

However, I never get a NextPageToken within the responses. Am I missing something? This is my code so far. Some key points:

  1. Althought there are more than +1000 items sharing the same prefix, I only get one response while looping through them in the enumerator.
  2. The NextPageToken in the Console.WriteLine statement is always null.
  3. Repeating the same call returns the same first 1000 objects over and over.
async IAsyncEnumerable<string> ListObjectsAsync(
    string prefix, [EnumeratorCancellation] CancellationToken cancelToken)
{
    var listObjectsOptions = new ListObjectsOptions
    {
        Fields = "items(name)"
    };

    var rawResponses = mGoogleClient.ListObjectsAsync(
        mBucketName,
        prefix,
        listObjectsOptions).AsRawResponses();

    using (var enumerator = rawResponses.GetEnumerator())
    {
        while (await enumerator.MoveNext(cancelToken))
        {
            foreach (var googleObject in enumerator.Current.Items)
                yield return googleObject.Name;
        }

        Console.WriteLine(enumerator.Current.NextPageToken);
    }
}

Thank you.

Upvotes: 2

Views: 971

Answers (1)

S_Luis
S_Luis

Reputation: 521

Apparently, I'm missing the nextPageToken field in the ListObjectsOptions used to make the request. Without specifying that field, the service won't return it - because of who knows why!

This code should work:

async IAsyncEnumerable<string> ListObjectsAsync(
    string prefix, [EnumeratorCancellation] CancellationToken cancelToken)
{
    var listObjectsOptions = new ListObjectsOptions
    {
        Fields = "items(name),nextPageToken"
    };

    var rawResponses = mGoogleClient.ListObjectsAsync(
        mBucketName,
        prefix,
        listObjectsOptions).AsRawResponses();

    using (var enumerator = rawResponses.GetEnumerator())
    {
        while (await enumerator.MoveNext(cancelToken))
        {
            foreach (var googleObject in enumerator.Current.Items)
                yield return googleObject.Name;
        }

        Console.WriteLine(enumerator.Current.NextPageToken);
    }
}

The nice thing is that you don't even have to use the NextPageToken explicitly. This is, you don't have to do something like this:

string token = null;
do
{
    var options = new ListObjectsOptions
    {
        Fields = "items(name),nextPageToken",
        PageToen = token
    };

    var rawResponses = mGoogleClient.ListObjectsAsync(
        mBucketName,
        prefix,
        listObjectsOptions).AsRawResponses();

    using (var enumerator = rawResponses.GetEnumerator())
    {
        while (await enumerator.MoveNext(cancelToken))
        {
            foreach (var googleObject in enumerator.Current.Items)
                yield return googleObject.Name;
        }

        token = enumerator.Current.NextPageToken;
    }
}
while (!string.IsNullOrEmpty(token));

...because the enumerator you got in rawResponses.GetEnumerator() will take care of using the token of a response to automatically fetch the next (if needed) while iterating them.

So the first piece of code is valid to iterate over +1000 objects in a single call.

Upvotes: 6

Related Questions