How can I set pageToken to get item lists from Google Cloud Storage via Java SDK?

Question

I want to set pageToken to get items stored at Google Cloud Storage. I'm using Google API Client Library for Java v1.19.x.

I have no idea to generate pageToken from file path(or file name).

2 files stored in bucket.

• my-bucket
  • /test.csv
  • /test2.csv

When I tried Google APIs Explorer with following parameters, I could get nextPageToken Cgh0ZXN0LmNzdg==.

And I found out that I can get test.csv string by decoding nextPageToken with base64.

• bucket: my-bucket
• pageToken:
• prefix: test
• maxResults: 1

{"kind": "storage#objects", "nextPageToken": "Cgh0ZXN0LmNzdg==", ...}

But How can I get Cgh0ZXN0LmNzdg== from test.csv?

Although I tried Base64 encoding, result didn't match.

import com.google.api.client.repackaged.org.apache.commons.codec.binary.Base64;

String lastFile = "test.csv"
String token = Base64.encodeBase64String(lastFile.getBytes());

String bucket = "my-bucket"
String prefix = "test"

Storage.Objects.List listObjects = client.objects().list(bucket);
listObjects.setPrefix(prefix);
listObjects.setPageToken(token);
long maxResults = 1;
listObjects.setMaxResults(maxResults);
do {
    Objects objects = listObjects.execute();
    List<StorageObject> items = objects.getItems();

    token = objects.getNextPageToken();
    listObjects.setPageToken(token);
} while (token != null);

Answer 1

The page token is indeed encoded as base64([LF] + length + name.bytes) as answered by @sakama. In addition to his answer, the length of the array is a varint which will answer how to encode names longer than 127 bytes. Example code:

    private static String pageToken(String name) {
        var encodedName = name.getBytes(StandardCharsets.UTF_8);
        // Use initial size of +3 (0x0a + 2 bytes for length of max 1024)
        var bos = new ByteArrayOutputStream(encodedName.length + 3);
        bos.write(0x0a);
        writeVarInt(bos, encodedName.length);
        bos.write(encodedName, 0, encodedName.length);
        return Base64.getEncoder().encodeToString(bos.toByteArray());
    }

    public static void writeVarInt(ByteArrayOutputStream bos, int value) {
        while ((value & ~0x7f) != 0) {
            bos.write((value & 0x7f) | 0x80);
            value >>>= 7;
        }
        bos.write(value);
    }

Maximum object name size is 1024

Answer 2

I know this question is already answered and is applied to Java, I'd like to mention that this question applies to PHP as well.

With the help of the approved post from sakama above I figured out a PHP version of his solution.

The PHP equivalent for generating the token is as follow:
base64_encode(pack('c', 0x0a) . pack('c', $path_string_length) . pack('a*', $path_string));

The byte pattern seems indeed (as sakama already mentioned) to be:
<line feed><line data length><line data>

Answer 3

I could get next token from file path string using following codes by myself.

How to get nextToken from path string

String nextToken = base64encode(0x0a + asciiCode + pathString)

asciiCode can be taken between 0x01(SOH) and 0x7f(DEL). It seems to depend on path length.

• my-bucket/
  • a/a(3byte) 0x03
  • a/ab(4byte) 0x04
  • test.txt(8byte) 0x08

Notice

If path length is longer than 1024 byte, another rule seems to apply. But I couldn't found out rules.

See also Object Name Requirements

import com.google.common.io.BaseEncoding;

String lastFile = "test.csv"
String token = base64Encode(lastFile);

String bucket = "my-bucket"
String prefix = "test"

Storage.Objects.List listObjects = client.objects().list(bucket);
listObjects.setPrefix(prefix);
listObjects.setPageToken(token);
long maxResults = 1;
listObjects.setMaxResults(maxResults);
do {
    Objects objects = listObjects.execute();
    List<StorageObject> items = objects.getItems();

    token = objects.getNextPageToken();
    listObjects.setPageToken(token);
} while (token != null);

private String base64Encode(String path) {
    byte[] encoding;
    byte[] utf8 = path.getBytes(Charsets.UTF_8);
    encoding = new byte[utf8.length + 2];
    encoding[0] = 0x0a;
    encoding[1] = new Byte(String.valueOf(path.length()));
    String s = BaseEncoding.base64().encode(encoding);
    return s;
}