CODEWITHSUNDEEP
pythongoogle-cloud-platform
red888
red888

Reputation: 31652

Where is the default max page size of GCP APIs documented?

Looking at the API reference I see there is no max page size described: https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/list

But does that mean I can make the page as big as I want? Right now I make it arbitrarily large:

from googleapiclient import discovery

discovery_service = discovery.build("dataflow", "v1b3", cache_discovery=False)

jobs = discovery_service.projects().locations().jobs().list(
    projectId=my_project,
    location="aggregated",
    pageSize=100000 # Is there any limit here?
).execute().get('jobs',[])

print(len(jobs))

I only have a few thousand jobs and it seems to be getting all of them, but I want to know if I can get away with this forever. Whenever I see a paginator for an API like this it always has some kind of limit to what it will return.

I'd rather set no limit like I am above because it's much simpler for what I need, but I don't trust the google docs. Is there really no limit at all, can this be confirmed?

Upvotes: 0

Views: 778

Answers (1)

Alex
Alex

Reputation: 864

Although there is not an specified boundary the pageSize definition is as follows:

integer

If there are many jobs, limit the response to at most this many. The actual number of jobs returned will be the lesser of max_responses and an unspecified server-defined limit.

There are other factors that affect the maximum number of items returned: The transfer limits of proxies and load balancers, the size of the individual items, network reliability, etc.

It might not be recommended to set a large value and instead use the default values with some exception scenarios. This to avoid the malfunctioning of an item that is extended with new data

Upvotes: 1

Related Questions