Google BigQuery error 401 msg:Unauthorized when processing large amount of data

Question

We are trying to retrieve a daily table from Google Big Query, which averages to around 40million rows per day through HTTP requests:

• Using a Service Account

• Since it is required in our case because of the amount of data, I am passing the allowLargeResults and providing the destinationTable parameters as described in [ https://cloud.google.com/bigquery/querying-data#largequeryresults ].

• The step above succeeds and I can see the total number of rows through reading the totalRows value from the JSON returned – this correctly corresponds to the total number of rows when I run the same query in BigQuery.

• We process the rows using paging, with getting the pageToken value for each page(we process around 3K rows per page as per the 10MB limit per page)

Having ran this process a few times whenever the total number of processed rows reaches near a million a NULL object is returned and we get the error 401 msg:Unauthorized.

Can you provide some more info on why this is happening? Having a look in the list of BigQuery's troubleshooting errors https://cloud.google.com/bigquery/troubleshooting-errors a 401 error is only described as an Authentication error, but I am pretty sure we succeed in authenticating with Google Big Query in the earlier steps.

Thanks for your time

Answer 1

The authorization token is valid for 3600 seconds, and you need to refresh the token in order to continue with the pagination.

Most libraries out there have dedicated method to refresh the token.