CODEWITHSUNDEEP
javascriptgoogle-cloud-platformgoogle-bigquerygoogle-cloud-functions
Bilel Abderrahmane BENZIANE
Bilel Abderrahmane BENZIANE

Reputation: 81

Google Cloud Function Error - PubSub to BigQuery

I created a Google Cloud Function with a Pub/Sub trigger (Triggered by a Stackdriver sink). Then it change the format of this data and save it to the BigQuery. 

const {BigQuery} = require('@google-cloud/bigquery');
const bigquery = new BigQuery();

const environment= process.env.ENVIRONMENT || 'Dev';

insertIntoBigQueryClient = async (locationObject) => {
    const metadata = locationObject.jsonPayload.metadata;

    const row = [{
        driverId: metadata.driverId,
        driverPhone: metadata.driverPhone,
        driverStatus: metadata.driverStatus,
        driverLocation: metadata.driverLocation.coordinates,
        timestamp: locationObject.timestamp
    }];
    // Insert data into a table
    return await bigquery
        .dataset(`YassirBackendLogging${environment}`)
        .table('DriverLocationStatus')
        .insert(row);
};


driverLocationStatusProcessing = async (pubSubEvent, context) => {
    try {
        const logObject = JSON.parse(Buffer.from(pubSubEvent.data, 'base64').toString());
        insertIntoBigQueryClient(logObject);
    } catch(error){
        console.error(error);
    }
};

// this part is only to have multi functions. one for each envirenment 
switch (environment) {
    case 'Prod' :
        exports.driverLocationStatusProcessingProd = async (pubSubEvent, context) => {
            await driverLocationStatusProcessing(pubSubEvent, context);
        };
        break;
    case 'Dev' :
        exports.driverLocationStatusProcessingDev = async (pubSubEvent, context) => {
            await driverLocationStatusProcessing(pubSubEvent, context);
        };
        break;

    default :
        exports.driverLocationStatusProcessingDev = async (pubSubEvent, context) => {
            await driverLocationStatusProcessing(pubSubEvent, context);
        };
        break;
}

And this is the cloud build code 

steps:
  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - functions
      - deploy
      - 'driverLocationStatusProcessing$_ENVIRONMENT'
      - '--set-env-vars'
      - ENVIRONMENT=$_ENVIRONMENT
      - '--trigger-topic'
      - 'DriverLocationStatus$_ENVIRONMENT'
      - '--runtime'
      - nodejs8
      - '--timeout=540'
    dir: 'driver-location-status'

now, this function is working perfectly but from time to time there are some errors out of no where. like the following :

Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. at GoogleAuth.getApplicationDefaultAsync (/srv/node_modules/google-auth-library/build/src/auth/googleauth.js:161:19) at at process._tickDomainCallback (internal/process/next_tick.js:229:7)

Error: function crashed out of request scope Function cannot be executed.

I hope i will have some feedback concerning that matter. Maybe something to do with async task?

Upvotes: 0

Views: 644

Answers (1)

Travis Webb
Travis Webb

Reputation: 15028

This looks like a potential logic error in your driverLocationStatusProcessing function:

try {
    ...
    return insertIntoBigQueryClient(logObject);
    // ^^ add return statement

I'm not sure if this is the cause of your issue, but your comments point to a potential race condition ("it happening like once in a million") and without that return, the await won't do what you expect it to.

This may also be relevant: Could not load the default credentials? (Node.js Google Compute Engine tutorial)

Upvotes: 2

Related Questions