CODEWITHSUNDEEP
google-app-enginegoogle-bigquery
Usman Ismail
Usman Ismail

Reputation: 18669

Running BigQuery load job on app engine gets stuck in pending state

I am trying to load data into BigQuery and the load job stays in the pending state indefinitely (I waited about 5 minutes). Is there something else I need to be doing to move the job to a running state? 

log.log(new LogRecord(Level.INFO, "Writing to big query table"));
JobConfigurationLoad loadConf = new JobConfigurationLoad();
loadConf.setDestinationTable(new TableReference().setProjectId(PROJECT_ID)
    .setDatasetId(datasetId).setTableId(TABLE_ID));
loadConf.setWriteDisposition("WRITE_APPEND");
loadConf.setSourceUris(Arrays.asList("gs://" + bucket + "/" + "something.json"));
JobConfiguration configuration = new JobConfiguration().setLoad(loadConf);
Job loadJob = new Job().setConfiguration(configuration);
Jobs.Insert insertData = bigQuery.jobs().insert(PROJECT_ID, loadJob);
Job insertResp = insertData.execute();

JobStatus status = insertResp.getStatus();
    while (!status.getState().equals("done")) {
        System.out.println(status.getState());
        status = insertResp.getStatus();
        Thread.sleep(10000);
}
...

Upvotes: 2

Views: 1628

Answers (2)

Jordan Tigani
Jordan Tigani

Reputation: 26617

Regarding the problem of not getting the latest job results, this code should work (if you add to what you had above:

// Insert the load job.
Job job = insertData.execute();  
JobId jobId = job.getJobId();

long startTime = System.currentTimeMillis();

while (!job.getStatus().getState().equals("DONE")) {
   // Pause execution for ten seconds before polling job status again
   Thread.sleep(10000);

   long elapsedTime = System.currentTimeMillis() - startTime;
   System.out.format("Job status (%dms) %s: %s\n", elapsedTime,
       jobId.getJobId(), job.getStatus().getState());       

   // Poll the server for job completion state.
   job = bigquery.jobs().get(projectId, jobId).execute();
}
if (job.getStatus().getErrorResult() != null) {
  // The job ended with an error.
  System.out.format("Job %s ended with error %s", job.getJobId(), 
      job.getStatus().getErrorResult().getMessage());
}

Upvotes: 1

Jordan Tigani
Jordan Tigani

Reputation: 26617

I've dug into this somewhat, and the issue is that you did 2300 import jobs to the same table, adding a new one every few seconds. The jobs started to get queued, since the new ones were arriving faster than they could be processed. This caused longer and longer pending times. It looks like the large number of jobs may have been unintentional, since they all seemed to be importing the same file to the same table. Incidentally, jobs all failed with an invalid_value error.

Note that this pattern only happened on 12/20. Other than this date I couldn't find any other jobs that you ran that lasted longer than 10 minutes.

Upvotes: 1

Related Questions