Mongoose findOneAndUpdate + upsert always replaces existing document

Question

I have a collection I want to upsert with findOneAndUpdate. In addition to that I have two fields (isHandled, isNotADuplicate) that should be:

• defaulted to 'false' upon insert
• left untouched upon update (e.g. isHandled stays 'true')

I have however found that

• isHandled, isNotADuplicate are always defaulted back to 'false'
• _id is also regenerated upon every update (I use a compound key to query the doc, not _id)

My Model

export const QuickbrainFindingSchema = new Schema<QuickBrainFindingDocument>({
    
    connectedApplicationType: { type: String, required: true, enum: ['jira'] },//e.g. jira
    clientKey: { type: String, required: true },//e.g. 135eb702-156c-3b67-b9d0-a0c97548xxxx
    
    //key
    projectKey: { type: String, required: true },//e.g. AL
    type: { type: String, required: true },
    doc1key: { type: String, required: true },//e.g. AL-7
    doc2key: { type: String, required: true },//e.g. AL-16
    
    //data
    calculationDate: { type: SchemaTypes.Date, default: Date.now },
    direction: { type: String, required: true },
    reasonAndMetric: { type: SchemaTypes.Mixed, reason: true },
    scoreSummary: { type: String, reason: true },

    isHandled: { type: SchemaTypes.Boolean, default: false },
    isNotADuplicate: { type: SchemaTypes.Boolean, default: false },

    similarityReference: { type: SchemaTypes.ObjectId, required: true, ref: "QuickbrainSimilarityMatrix" }

}, {
    //options
});

QuickbrainFindingSchema.index(
    { connectedApplicationType: 1, clientKey: 1, project: 1, doc1key: 1, doc2key: 1, type: 1 },
    { unique: true, name: "compoundKey" }
);

export const QuickbrainFindingModel = model<QuickBrainFindingDocument>("QuickbrainFinding", QuickbrainFindingSchema);

My Code

public async addFinding(
        projectKey: string,
        doc1key: string,
        doc2key: string,
        type: ET_FindingType
        , data: QuickbrainFindingData): Promise<QuickbrainFinding> {

        let keyFull: QuickbrainFindingKey = {
            connectedApplicationType: this.connectedApplicationType,
            clientKey: this.clientKey,
            projectKey: projectKey,
            doc1key: doc1key,
            doc2key: doc2key,
            type: type
        };

        let insertObj: QuickbrainFinding = <QuickbrainFinding><unknown>{};
        Object.assign(insert, keyFull);
        Object.assign(insert, data);

        delete (<any>insertObj).isHandled;
        delete (<any>insertObj).isNotADuplicate;

        return new Promise<QuickbrainFinding>(function (ok, nok) {

                QuickbrainFindingModel.findOneAndUpdate(
                    keyFull, { $set: insertObj},
                    {
                        runValidators: true,
                        upsert: true,
                        setDefaultsOnInsert: true,
                        new: true,
                        omitUndefined: true,//I think only available for findAndReplace(..)
                    })
                .lean().exec(function (err, result) {
                    if (err) {
                        nok(err);
                    }
                    else
                        ok(result)
                });

        });

    }

Mongoose Debug Output

        quickbrainfindings.findOneAndUpdate(
            {
                connectedApplicationType: 'jira',
                clientKey: '135eb702-256c-3b67-b9d0-a0c975487af3',
                projectKey: 'ITSMTEST',
                doc1key: 'ITSMTEST-7',
                doc2key: 'ITSMTEST-10',
                type: 'Email'
            },
            {
                '$setOnInsert':
                    { __v: 0, isHandled: false, isNotADuplicate: false, _id: ObjectId("60789b02c094eb3ef07d2929") },
                '$set': {
                    connectedApplicationType: 'jira',
                    clientKey: '135eb702-256c-3b67-b9d0-a0c975487af3', projectKey: 'ITSMTEST', doc1key: 'ITSMTEST-7', doc2key: 'ITSMTEST-10', type: 'Email',
                    calculationDate: new Date("Thu, 15 Apr 2021 19:58:58 GMT"),
                    direction: '2', scoreSummary: '100.0%',
                    similarityReference: ObjectId("60789b029df2079dfa8aa15a"),
                    reasonAndMetric: [{ reason: 'Title Substring', metricScore: '100%' },
                    { reason: 'Title TokenSet', metricScore: '54%' }, { reason: 'Description TokenSet', metricScore: '100%' }]
                }
            },
            {
                runValidators: true, upsert: true, remove: false, projection: {},
                returnOriginal: false
            }
        )

What happens

Existing documents are found, but when they are updated I'm confused that:

• _id is regenerated
• isHandled and isNotADuplicate are reset to 'false' (although insertObj does not contain them)
• When looking at the debug output I can see that the new _id is the one fron $setOnInsert, which confuses the heck out of me, since the selector works

Notable

• keyFull is used to query the existing document, it does not contain _id;
• delete (<any>insertObj).isHandled <- the object used for $set does NOT contain isHandled

Answer 1

This is embarrasing to admit, but thanks to Joe I have found the problem.

Before every findOneAndUpdate / Upsert I had a delete statement removing the existing documents Pipeline:

• Delete old documents
• Calculate new documents
• Upsert new documents -> always resulted in Insert

            let matchAnyDoc = this.filterForDocKeyAny(projectKey, docKeyAny, findingType);
            matchAnyDoc.forEach(async (condition) => {
                QuickbrainFindingModel.deleteMany(condition).exec(function (err, res) {
                    if (err) {
                        nok(err);
                    } else {
                        ok();
                    }
                });

            }, this);