CODEWITHSUNDEEP
google-bigquery
chandan singh
chandan singh

Reputation: 49

How to use insertId with big query streaming insert for each rows?

How to use insertId with Bigquery streaming insert for each row by node.js 

const bigquery = new BigQuery({
  projectId: projectId,
});

// Inserts data into a table
await big query
  .dataset(datasetId)
  .table(tableId)
  .insert(rows);
console.log(`Inserted ${rows.length} rows`);

Upvotes: 3

Views: 2225

Answers (2)

Joseph Lust
Joseph Lust

Reputation: 19975

This feature is poorly documented in the Node.js docs (by which I mean undocumented 😭). However, there is an example in the unit tests for the Node.js library here and here.

The part I had to dig into the unit tests to find was {raw: true}.

Here is a full working example:

const {BigQuery} = require('@google-cloud/bigquery');
const bigquery = new BigQuery();

const rows = [
  {
    insertId: 'my_insert_id',
    json: {foo: 'bar', baz: 'bax'}
  }
];

await bigquery
  .dataset('my_dataset')
  .table('my_table')
  .insert(rows, {raw: true});

Upvotes: 3

ch_mike
ch_mike

Reputation: 1576

The insertId is included on the InsertionRow structure, which is:

{
  "insertId": string,
  "json": {
    object
  }
}

There is a nodejs example here, but basically:

const rows = [
{
  insertId: '1',
  json: {
    INSTNM: 'Motion Picture Institute of Michigan',
    CITY: 'Troy',
    STABBR: 'MI'
  }
},
//...
];

As mentioned on the document Graham shared:

To help ensure data consistency, you can supply insertId for each inserted row. BigQuery uses the insertId property to de-duplicate your data on a best effort basis.

Upvotes: 1

Related Questions