CODEWITHSUNDEEP
javascriptgoogle-cloud-firestoregoogle-cloud-functionsgoogle-api-nodejs-clientgoogle-admin-sdk
Muloni Bryan
Muloni Bryan

Reputation: 11

Importing large data from BigQuery to Firestore using Cloud Functions

I am working on importing large data from a BigQuery dataset table into Firestore using Cloud Functions, all is well with small data tables of about 500 records. But the problem is with large data, for example, a table of about 40,000 records. From Firebase quota limit of 500 documents, they recommend using Batched writes which I have used but it still only inserts 500 documents, in logs, I see more than 500 documents have been created(I have logged auto-generated ID's) but I can not see them in Firestore. I have configured the enter code heretimeout to 540 seconds. Help me look at my code and see where I am going wrong. Thanks

const {BigQuery} = require("@google-cloud/bigquery");
const firebaseConfig =  require('firebase-admin');

firebaseConfig.initializeApp();

const firestoreDb = firebaseConfig.firestore();

var batchCommits = [];

var batch = firestoreDb.batch();

var counter = 0;

exports.fetchAdminTwo = (req,res) =>{
 var bigQuery = new BigQuery({ projectId: 'XXXXXXXXXXX' });

 var sqlStatement = 'SELECT id, id_admin1, description_en, description_fr, description_es, latitude, longitude FROM `fao-empres-re.empres_data.admin2`;';

 console.log('Triggering bigquery to execute sql statement');

 return bigQuery.query({query:sqlStatement}).then(results =>{
     var adminList = results[0];
     console.log('Query results:',adminList);

     res.send(adminList);

     adminList.forEach((admin,i) =>{
         var collectionRef = firestoreDb.collection('ref-admins-two').doc();
         batch.set(collectionRef,admin);
         console.log('Doc ID:',collectionRef.id,'Count:',counter++);
         if ((i + 1) % 500 === 0) {
             console.log(`Batch count: ${i + 1}`);
             batchCommits.push(batch.commit());
             batch = firestoreDb.batch();
             }
     });
     batchCommits.push(batch.commit());
     return Promise.all(batchCommits);

}).catch(error =>{
    console.log("Query error:",error);
    });

}

Upvotes: 1

Views: 553

Answers (1)

Kamal
Kamal

Reputation: 61

In your code you are not resetting batchCommit to empty array before taking in next 500 batch. Modify your code something like this.

if ((i + 1) % 500 === 0) {
             console.log(`Batch count: ${i + 1}`);
             batchCommits.push(batch.commit());
             batch = firestoreDb.batch();
             batchCommits = [];
             }
=

Upvotes: 1

Related Questions