Mongo $lookup $pipeline with several $expr parameters not returning correct data in $lookup results

Question

I have two mongo collections one containing appointment reminders and the other containing notifications. I'm attempting to return results with all archive: false reminders in a given branchId / clinic_id and include their acknowledged: false notifications. Also, I want to make sure the appointments show up in the results regardless if they have notifications or not.

I need to 'join' the collections by branchId aka clinic_id and then create an array of any unacknowledged notifications for each resulting appointment reminder.

I'm returning correct appointments, but the notif array is not getting filtered by matching patientId / patient_id. Each reminder seems to contain the exact same notif array. Other than this everything else seems to be correct. So, my question is how can I ensure the notif array only contains a patientId that matches the patient_id value of the reminder?

Truncated Appointment Reminder Schema:

      {
      time: {
        type: Date,
        required: true
      },
  patient_id: {
    type: String,
    required: true
  },
      status: {
        type: String,
        default: 'unconfirmed'
      },
      archive: {
        type: Boolean,
        default: false
      },
      branch: [
    // Incorrectly setup as an array rather than an object. 
    // Can $unwind as there is only ever one item in the array
        {
          name: {
            type: String,
            required: true
          },
          clinic_id: { // aka branchId
            type: String,
            required: true
          }
        }
      ]
    }

Notification Schema:

{
  branchId: { type: String, required: true }, // aka clinic_id
  patientId: { type: String, required: true },
  acknowledged: { type: Boolean, default: false },
  date: { type: Date, default: Date.now }
}

Aggregation Query:

[
  { $match: { 'branch.0.clinic_id': '1', archive: false } },
  { $unwind: '$branch' },
  {
    $lookup: {
      from: 'notifications',
      let: { clinic_id: '1', patient_id: '$patientId' }, //<-- issue with patient_id?
      pipeline: [
        {
          $match: {
            $expr: {
              $and: [
               { $eq: ["$patientId", "$$patientId"] }, <-- errors $$patientId unknown value. $$patient_id returns 0 results.
                { $eq: ['$branchId', '$$clinic_id'] },
                { $eq: ['$acknowledged', false] }
              ]
            }
          }
        }
      ],
      as: 'notif'
    }
  }
]

Example Output with comments on desired output and incorrect output I've experienced:

  {
  patient_id: '1',
  time: '2019-05-29T11:00:00.000Z',
  status: 'unconfirmed',
  archive: false,
  branch: [
    {
      name: 'Example location',
      clinic_id: '100',
    }
  ],
  notif: [
  {
    // This is correct
    branchId: '100', // branchId matches clinic_id
    patientId: '1',  // patientId matches contacts patient_id
    acknowledged: false, // notification is unacknowledged
    date: '2019-05-18T16:18:05.480Z'
  },
  {
   // This is not correct
    branchId: '100', 
    patientId: '2', // PatientId does not match patient_id of reminder
    acknowledged: false,
    date: '2019-05-20T16:18:05.480Z'
  }
  ]
}

Answer 1

With the help of tom slabbaert, this solved the issue:

[
        { $match: { 'branch.clinic_id': '1', archive: false } },
        { $unwind: '$branch' },
        {
          $lookup: {
            from: 'notifications',
            let: { clinic_id: '1', patient_id: '$patient_id' }, // <-- changed here
            pipeline: [
              {
                $match: {
                  $expr: {
                    $and: [
                      { $eq: ["$$patient_id", "$patientId"]}, // <-- changed here
                      { $eq: ['$branchId', '$$clinic_id'] },
                      { $eq: ['$acknowledged', false] }
                    ]
                  }
                }
              }
            ],
            as: 'notif'
          }
        }
      ]

Answer 2

First you do have to use $$patient_id, thats the correct syntax for using a lookup variable.

The reason why your getting 0 results with the correct syntax is (i'm assuming as you didnt share the full schema) type different.

Notice that is patientId in your notification schema is defined:

patientId: { type: String, required: true }, which is typed String.

From the "desired" output schema you shared at the end:

{ patientId: 1, ... }

It looks like your patientId is defined as a Number, hence the reason don't get a match between the documents.