CODEWITHSUNDEEP
google-cloud-dataflowgoogle-cloud-functions
rish0097
rish0097

Reputation: 1094

Confirm success of a dataflow job in Cloud functions

Is there a way for cloud functions to confirm whether a dataflow job has succeeded or not??

Cloud function that I tried:

const google = require('googleapis');

exports.statusJob = function(event, callback) {
 const file = event.data;
 if (file.resourceState === 'exists' && file.name) {
     console.log(file.name);
   console.log(event.data);
   google.auth.getApplicationDefault(function (err, authClient, projectId) {
     if (err) {
       throw err;
     }

     if (authClient.createScopedRequired && authClient.createScopedRequired()) {
       authClient = authClient.createScoped([
         'https://www.googleapis.com/auth/cloud-platform',
         'https://www.googleapis.com/auth/userinfo.email'
       ]);
     }

     const dataflow = google.dataflow({ version: 'v1b3', auth: authClient });

     dataflow.projects.jobs.get({
       projectId: 'my-project-id',
       resource: {
         jobId: 'some_number'
       }
     }, function(err, response) {
       if (err) {
         console.error("problem running dataflow template, error was: ", err);
       }
       console.log("Dataflow template response: ", response);
       callback();
     });

   });
 }
};

Package JSON:

{
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "googleapis": "^18.0.0"
  }
}

The above thing worked for me perfectly ONCE. The response that I got was:

Dataflow template response: { id: 'some_number', projectId: 'my-project-id', name: 'cloud-fn', type: 'JOB_TYPE_BATCH', environment: { userAgent: { name: 'Google Cloud Dataflow SDK for Java', support: [Object], 'build.date': '2017-05-23 19:46', version: '2.0.0' }, version: { major: '6', job_type: 'JAVA_BATCH_AUTOSCALING' } }, currentState: 'JOB_STATE_DONE',........

Then it gave an error each time after that saying:

problem running dataflow template, error was: Error: Missing required parameters: jobId at createAPIRequest (/user_code/node_modules/googleapis/lib/apirequest.js:110:14) at Object.get (/user_code/node_modules/googleapis/apis/dataflow/v1b3.js:670:16) at /user_code/index.js:22:29 at callback (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/auth/googleauth.js:42:14) at /user_code/node_modules/googleapis/node_modules/google-auth-library/lib/auth/googleauth.js:289:13 at _combinedTickCallback (internal/process/next_tick.js:73:7) at process._tickDomainCallback (internal/process/next_tick.js:128:9)

Does anyone know anything about this?

Thanks

Upvotes: 2

Views: 1421

Answers (2)

Karups
Karups

Reputation: 11

The payload is wrong. It should be like below.

 dataflow.projects.jobs.get({
   projectId: 'my-project-id',
   jobId: 'some_number'
 }, function(err, response) {
   if (err) {
     console.error("problem running dataflow template, error was: ", err);
   }
   console.log("Dataflow template response: ", response);
   callback();
 });

The above code is fetching the job details successfully with Node.js 8 (beta).

Upvotes: 0

Lara Schmidt
Lara Schmidt

Reputation: 309

You can use the Dataflow CLI to determine if a job failed or succeeded. It lets you list jobs and also check their failed/success/running/cancelled status.

Specifically, to check the state of a single job, you may run:

gcloud beta dataflow jobs describe <JOB_ID>

For more info check the docs:

https://cloud.google.com/dataflow/pipelines/dataflow-command-line-intf

Upvotes: 1

Related Questions