How to make "pairing mode" (whitelist not working) with Arduino BLE and ESP32

Question

I am trying to make a "pairing mode" with my ESP32 BLE server, where I want the previously paired device to be able to join whenever no devices are connected, but when the user presses a button every device should be able to join and whoever joins will become the new "paired" device.

I tried using the whitelist, but even when I use setScanFilter(true, false) it shows up for every device in nRF connect, even ones that haven't been paired, and also allows them to connect. When I use setScanFilter(true, true) it still shows up for every device, but doesn't allow any to connect.

I tried using the ESP_BLE_SEC_ENCRYPT_MITM but that allows anyone to connect, it just pairs in the settings app now. It has no effect on the whitelist and the device still shows up for everyone.

If I stop advertising even paired devices can't connect.

How can I go about implementing this? Here is my whitelist code (relevant sections):

  preferences.begin("ble_data", false);
  String storedMAC = preferences.getString("phoneMAC", "");
  if (storedMAC.length() > 0) {
    phoneMAC = BLEAddress(storedMAC.c_str());
    BLEDevice::whiteListAdd(phoneMAC);
  } else {
    phoneMAC = BLEAddress("");
  }

  void onConnect(BLEServer *pServer, esp_ble_gatts_cb_param_t *param) {
    if (!BLEAddress(param->connect.remote_bda).equals(phoneMAC)) {
      if (!phoneMAC.equals(BLEAddress(""))) {
        BLEDevice::whiteListRemove(phoneMAC);
      }

      // Update preferences
      preferences.putString(
          "phoneMAC", BLEAddress(param->connect.remote_bda).toString().c_str());
      phoneMAC = BLEAddress(param->connect.remote_bda);
      BLEDevice::whiteListAdd(phoneMAC);
    }
    BLEAdvertising *pAdvertising = pServer->getAdvertising();
    pAdvertising->stop();
    advertisingMode = OFF;
  }