Reputation: 372
I am using Java SDK jars for BigQuery data fetching. From BigQuery I am fetching a detailed Usage Cost report for Invoice month. In my account, there are more resources so the Bill value is also high. The total rows present in the BigQuery for the particular invoice month is 19680970. so I used the below approach like querying for the entire invoice month once so that Big Query will have a temporary table by that results will be quicker and cost efficient.
Unfortunately, the process took more than a day, so that while querying from the nextPage token as per the below code getting Table not found. I think internally nextPage token points to the temporary table. As per Google documentation Temporary table expire after 24 hours. It seems to be not handled at Google side by changing nextPage token.
My code is
String query = "SELECT * FROM `Sample-3806.Usage.gcp_billing_export_resource_v1_01HYD5_5UDWC_836E8` where invoice.month="202310";
TableResult tableResult = this.bigQueryExecutor.executeBigQuery(query);
if(tableResult!=null)
{
do
{
if(isNextPageAvailable && tableResult!=null)
{
tableResult = tableResult.getNextPage();
isNextPageAvailable = false;
}
if(tableResult!=null)
{
Iterable<FieldValueList> itr = tableResult.getValues();
for(FieldValueList valueList : itr)
{
// Code logics
}
isNextPageAvailable = tableResult.hasNextPage();
}
}while(isNextPageAvailable);
}
Getting exception like ::
{
"code": 404,
"errors": [
{
"domain": "global",
"message": "Not found: Table Sample-3806.Usage.gcp_billing_export_resource_v1_01HYD5_5UDWC_836E8:_d5733737d9bad3beded5b.35no6nc1575fccadb9fc743d158beda8a28fca489",
"reason": "notFound"
}
],
"message": "Not found: Table Sample-3806.Usage.gcp_billing_export_resource_v1_01HYD5_5UDWC_836E8:_d5733737d9bad3beded5b.35no6nc1575fccadb9fc743d158beda8a28fca489",
"status": "NOT_FOUND"
}
What changes are required to fix this by cost efficient.
Upvotes: 0
Views: 153