Update one, insert if not exists and modify only when a condition is true MongoDB

Question

I am writing a program in Node.js using MongoDB and I want to update a database when certain files are modified. Inside the database I store the mtime of each database entry and I want to compare this to the mtime of the file to see if the database entry is out of date. Here is my code:

function updateArticles(articles) {
    const dir = "articles";
    const files = fs.readdirSync(dir);

    files.forEach(
        (file) => {
            const fpath = path.join(dir, file);
            const fmtime = fs.statSync(fpath).mtimeMs;
            articles.updateOne(
                //refresh content and mtime if the text file has been modified
                {
                    _id: file,
                    mtime: {$lte: fmtime}
                },
                {
                    $set: {content: htmlifySync(fpath), mtime: Date.now(), title: titleSync(fpath)},
                    //btime is only set once
                    $setOnInsert: {btime: Date.now()}
                },
                {upsert: true}
            )
        });
}

When I set {upsert: true} I get a duplicate key error. When I set {upsert: false} new entries are not added. It works if I remove the second condition in my query but then it is running htmlifySync and titleSync for every entry which is expensive and unneccessary.

I think the issue is that mongoDB is trying to insert a new entry which meets both query conditions which means it tries to insert a new entry with the same id. I want mtime: {$lte: fmtime} to be a filter for the update but I don't want it to be made true by the update.

To clarify:

• If there is a DB entry for the file:
  • If the file has been modified since the last DB update I want to update it
  • If the file hasn't been modified since the last DB update I want to leave it unchanged
• If there isn't a DB entry for the file I want to create one

Answer 1

I've realised that what I really want to avoid is evaluating htmlifySync for documents that don't need updating. I think the only way to do that is to first findOne and then insertOne or updateOne as the only way to stop htmlifySync and titleSync being evaluated each time is to put them in if statements.

articles.findOne({_id: file}, {mtime: 1}).then(
    entry => {
        if (!entry) {
            articles.insertOne(
                {
                    _id: file,
                    mtime: Date.now(),
                    title: titleSync(fpath),
                    content: htmlifySync(fpath),
                    btime: Date.now()
                }
            ).then((res) => {
                console.log(res.res)
            })
        } else if (entry.mtime < fmtime) {
            console.log(file);
            articles.updateOne(
                {_id: file},
                {
                    $set: {
                        mtime: Date.now(),
                        title: titleSync(fpath),
                        content: htmlifySync(fpath)
                    }
                }
            ).then((res) => {
                console.log(res.res)
            })
        }
    }
)

Answer 2

The update operation will try to insert a new document if there are no matches. Since you will have no match if the file has not been updated, it tries to insert a new one, triggering the error.

Possible solutions:

• Catch the error and ignore it
  The duplicate key error means that the given file already exists in the database but the file time is not greater than the stored time, so no action necessary.
• Use a conditional assignment in the update If you are using MongoDB 4.2, you can use aggregation expressions in the update.

articles.updateOne(
                {
                    _id: file
                },
                [{
                    $set: {
                            content: htmlifySync(fpath), 
                            mtime: {$cond:[{$lt:["$mtime",fmtime]},Date.now(),"$mtime"]} 
                            title: titleSync(fpath),
                            btime: {$cond:[{$eq:[{$type:"$btime"},"missing"]},Date.now(),"$btime"]}
                    }
                }],
                {upsert: true}
            )

This uses $cond to only set btime if it doesn't already exist, and only set mtime if the stored value is less than fmtime, and takes advantage of the fact that MongoDB will not actually process a write if the update sets the same values that already exist.