CODEWITHSUNDEEP
javagoogle-bigquery
Roberto G.
Roberto G.

Reputation: 171

BigQuery - How to iterate over results using java API

I am trying to download a query result of 90k rows (this amount states the web UI and the resulting object) but I obtain more than 130k items in the iterator as follows:

QueryRequest queryRequest = QueryRequest
                                        .newBuilder("......")
                                        .setUseLegacySql(true)
                                        .build();
QueryResponse response = bigquery.query(queryRequest);
QueryResult result = response.getResult();
Integer c = 0;

while(result != null){
          Iterator<List<FieldValue>> iter = result.iterateAll();
          while(iter.hasNext()){
              iter.next();
              c++;
            }    
          result = result.getNextPage();
}

At the end of the reading is about 130K but reuslt.getTotalRows() contains 90K.

Do you have any idea on what I'am doing not correctly?

Upvotes: 2

Views: 2954

Answers (1)

P&#233;ter Farkas
P&#233;ter Farkas

Reputation: 31

From JavaDoc, Page.iterateAll() returns results from all pages.

You get all results (from the first page), then advance to the next page, where you get all results again (minus the first page). Setting the page size to a smaller number actually increases the results you get.

Your code should look like this:

QueryRequest queryRequest = QueryRequest
                                    .newBuilder("......")
                                    .setUseLegacySql(true)
                                    .build();
QueryResponse response = bigquery.query(queryRequest);
QueryResult result = response.getResult();
Integer c = 0;

Iterator<List<FieldValue>> iter = result.iterateAll();
while(iter.hasNext()){
    iter.next();
    c++;
}

Upvotes: 3

Related Questions