CODEWITHSUNDEEP
node.jsfirebasegoogle-cloud-firestorefirebase-admin
今際のアリス
今際のアリス

Reputation: 354

Batch 500 writes into firestore loop from json file

Using some inspiration I got from this thread and reply I tried to get my loop working which is to write into firestore in batches. But somehow I only can only update 1 document even if I can see I iterate through different values from my array. I load data into an array and work from there.

const db = admin.firestore();
const jsonStream = StreamArray.withParser();

let arr = []
jsonStream.on('data', ({ key, value }) => {
    arr.push(value);
});

jsonStream.on('end', () => {

    var counter = 0;
    var commitCounter = 0;
    var batches = [];

    arr.forEach((a, ind) => {
        batches[commitCounter] = db.batch();
        if (counter <= 498) {
            var thisRef = db.collection('Testing').doc(a.id);
            console.log("id")
            console.log(a.id);
            batches[commitCounter].set(thisRef, { ...a });
            counter = counter + 1;
        } else {
            counter = 0;
            commitCounter = commitCounter + 1;
            batches[commitCounter] = db.batch();
        }
    })
    for (var i = 0; i < batches.length; i++) {
        if(i==0)
        {
            console.log(batches[0])
        }
        batches[i].commit().then(function () {
            console.count('wrote batch');
        });
    }
});

const filename = path.join(__dirname, 'mydata.json');
fs.createReadStream(filename).pipe(jsonStream.input);

Upvotes: 0

Views: 347

Answers (1)

Hiranya Jayathilaka
Hiranya Jayathilaka

Reputation: 7438

Following line gets executed on each iteration, which essentially "resets" your batch on each round:

batches[commitCounter] = db.batch();

So at the end each of your batches will only contain one document write.

Upvotes: 2

Related Questions