Python based central program using D-Bus module unable to discover *some* BLE peripherals on Raspberry PI 4

Question

Summary

A python based BLE Central program, running on Raspberry PI 4, is unable to discover specific BLE peripherals. The same program when run on a Linux machine is able to discover and connect to the the specific BLE peripherals.

The Linux machine was the development platform for the above program and the target machine for this program is a Raspberry Pi 4 (8GB Ram)

What has been done so far.

In order to find answers I have done the following:

1. Purpose: to elimnate any issues with BLE Peripheral: The specific BLE peripherals mentioned above are a custom hardware based on Nordic's nRF52832 and has been tested independently using nRF Connect as well as another Android app for discovery, connection and data transfer

2. Purpose: to see if other bluetooth tools are able to discover the specific BLE peripherals: hcitool and btmgmt was used to discover the specific BLE devices and both successfully discovered the devices. This was done both on Linux as well as Raspberry PI and the results were same - both succeeded in discovering the devices. Next bluetoothctl was used to scan but it failed to discover the BLE devices that were discovered by hcitool and btmgmt. The same was true on Linux as well as Raspberry pi

However my python program based on D-Bus module is able to discover these BLE devices on Linux but not on Raspberry pi.

Version Numbers

Linux Kernel : 5.4.0-126-generic, bluetoothctl : 5.53, bluetoothd : 5.53, hcitool : 5.53, btmgmt : 5.53, python : 3.9.5

Raspberry PI Kernel : 5.15.61-v, bluetoothctl : 5.65, bluetoothd : 5.65, hcitool : 5.55, btmgmt : 5.55, python : 3.9.2

Grateful for some way forward or cues to solve this.

UPDATE 3/Oct/2022:

I further checked the bluetooth service status using sudo service bluetooth status and found there was a difference between Linux and Raspberry status but am unclear as to what this means especially the one relating to last 6 lines of status relating to raspberry pi

On Linux

sudo service bluetooth status

bluetooth.service - Bluetooth service
     Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-10-03 10:42:15 IST; 4h 10min ago
       Docs: man:bluetoothd(8)
   Main PID: 998 (bluetoothd)
     Status: "Running"
      Tasks: 1 (limit: 18985)
     Memory: 4.2M
     CGroup: /system.slice/bluetooth.service
             └─998 /usr/lib/bluetooth/bluetoothd --experimental

Oct 03 10:42:14 udu-Inspiron-7559 systemd[1]: Starting Bluetooth service...
Oct 03 10:42:15 udu-Inspiron-7559 bluetoothd[998]: Bluetooth daemon 5.53
Oct 03 10:42:15 udu-Inspiron-7559 systemd[1]: Started Bluetooth service.
Oct 03 10:42:15 udu-Inspiron-7559 bluetoothd[998]: Starting SDP server
Oct 03 10:42:15 udu-Inspiron-7559 bluetoothd[998]: Bluetooth management interface 1.14 initialized
Oct 03 10:42:36 udu-Inspiron-7559 bluetoothd[998]: Endpoint registered: sender=:1.75 path=/MediaEndpoint/A2DPSink/sbc
Oct 03 10:42:36 udu-Inspiron-7559 bluetoothd[998]: Endpoint registered: sender=:1.75 path=/MediaEndpoint/A2DPSource/sbc

On Raspberry Pi

sudo service bluetooth status

     Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-10-02 19:22:56 IST; 19h ago
       Docs: man:bluetoothd(8)
   Main PID: 884 (bluetoothd)
     Status: "Running"
      Tasks: 1 (limit: 8986)
        CPU: 446ms
     CGroup: /system.slice/bluetooth.service
             └─884 /usr/libexec/bluetooth/bluetoothd --experimental

Oct 02 19:22:56 kosha bluetoothd[884]: Battery Provider Manager created
Oct 02 19:22:56 kosha bluetoothd[884]: Adv Monitor Manager created with supported features:0x00000000, enabled features:0x00000000, max number of supported monitors:32, max number of supported patterns:16
Oct 02 19:22:56 kosha bluetoothd[884]: Endpoint registered: sender=:1.27 path=/MediaEndpoint/A2DPSink/sbc
Oct 02 19:22:56 kosha bluetoothd[884]: Endpoint registered: sender=:1.27 path=/MediaEndpoint/A2DPSource/sbc
Oct 03 11:42:55 kosha bluetoothd[884]: Path / reserved for Adv Monitor app :1.51
Oct 03 11:45:14 kosha bluetoothd[884]: Adv Monitor app :1.51 disconnected from D-Bus
Oct 03 11:45:19 kosha bluetoothd[884]: Path / reserved for Adv Monitor app :1.52
Oct 03 13:58:28 kosha bluetoothd[884]: Adv Monitor app :1.52 disconnected from D-Bus
Oct 03 14:10:52 kosha bluetoothd[884]: Path / reserved for Adv Monitor app :1.72
Oct 03 14:26:38 kosha bluetoothd[884]: Adv Monitor app :1.72 disconnected from D-Bus

UPDATE 7/Oct/2022

A skeletal code replicating the behaviour of the python App below, however not having access to the BLE peripherals will prevent you from seeing the problem (i.e. Works fine on Linux 20.04 but not on Raspberry PI)

#!/usr/bin/python
# SPDX-License-Identifier: LGPL-2.1-or-later


import dbus
import dbus.mainloop.glib
from gi.repository import GLib
import sys, signal


BLUEZ_SERVICE_NAME = "org.bluez"
BLUEZ_NAMESPACE = "/org/bluez/"
DBUS_OM_PROPERTIES="org.freedesktop.DBus.Properties"
DBUS_OM_IFACE = 'org.freedesktop.DBus.ObjectManager'

ADAPTER_INTERFACE = BLUEZ_SERVICE_NAME + ".Adapter1"
DEVICE_INTERFACE = BLUEZ_SERVICE_NAME + ".Device1"

relevant_ifaces = ( ADAPTER_INTERFACE, DEVICE_INTERFACE )

# looking for device with these addresses
addrs = ["D0:5F:64:52:00:01","D0:5F:64:52:13:45"]
log_detail = 1
log_added = 1
log_removed = 1
not_of_interest = 1

addrs = [x.replace(":","_") for x in addrs ]
print(f"Devices with mac address {addrs} are of interest")
def property_changed(interface, changed, invalidated, path):
    global devices, current_path
    iface = interface[interface.rfind(".") + 1:]
    #print(f"[{interface}]")
    if "Device1" in interface:
        # check if it is device of interest - get "_" separated mac address from path
        _devaddr = path[path.rfind("/")+5:]
        if _devaddr in addrs:
            pkeyval(changed,f"CHG: {_devaddr} ")
        else:
            if not_of_interest:
                print(f"CHG: Device Not of interest : {_devaddr}")
    else:
        if not_of_interest:
            print(f"CHG: Interface not of interest: {iface}")

def interfaces_added(path, interfaces):
    for iface, props in interfaces.items():
        if not(iface in relevant_ifaces) or log_added == 0:
            continue
        if iface in ADAPTER_INTERFACE:
            print(f"ADD: Adapter [{path}]")
        else:
            # its a device interface : get name and address from props and path respectively
            _devaddr = path[path.rfind("/")+5:]
            _name = "None"
            if "Name" in props.keys():
                _name = props['Name']
            print(f"ADD: Device {_devaddr} Name: {_name}" )

def interfaces_removed(path, interfaces):
    for iface in interfaces:
        if not(iface in relevant_ifaces) or log_removed == 0:
            continue
        if iface in DEVICE_INTERFACE:
            _devaddr = path[path.rfind("/")+5:]
            print(f"DEL: Device {_devaddr}")

def pkeyval(d,title=None):
    ''' given d dbus dict print key val : Warning no checking for types'''
    for k,v in d.items():
        print(f"{title} {dbus_to_python(k)} = {dbus_to_python(v)}")
    
def find_adapter():
    objects=get_managed_objects()
    for o, props in objects.items():
        if GATT_MANAGER_INTERFACE in props.keys():
            return o

def signal_handler(sig, frame):
    #global buff, fname
    global devices
    try:
        mainloop.quit()
    except Exception as e:
        log_msg.error(e)    
    print("Exiting")
    sys.exit(0)


def get_managed_objects():
    bus = dbus.SystemBus()
    manager = dbus.Interface(bus.get_object(BLUEZ_SERVICE_NAME, "/"),DBUS_OM_IFACE)
    return manager.GetManagedObjects()

def find_adapter(pattern=None):
    return find_adapter_in_objects(get_managed_objects(), pattern)

def find_adapter_in_objects(objects, pattern=None):
    bus = dbus.SystemBus()
    for path, ifaces in objects.items():
        adapter = ifaces.get(ADAPTER_INTERFACE)
        if adapter is None:
            continue
        if not pattern or pattern == adapter["Address"] or \
                            path.endswith(pattern):
            obj = bus.get_object(BLUEZ_SERVICE_NAME, path)
            return dbus.Interface(obj, ADAPTER_INTERFACE)
    raise Exception("Bluetooth adapter not found")

def dbus_to_python(data):
    if isinstance(data, dbus.String):
        data = str(data)
    if isinstance(data, dbus.ObjectPath):
        data = str(data)
    elif isinstance(data, dbus.Boolean):
        data = bool(data)
    elif isinstance(data, dbus.Int64):
        data = int(data)
    elif isinstance(data, dbus.Int32):
        data = int(data)
    elif isinstance(data, dbus.Int16):
        data = int(data)
    elif isinstance(data, dbus.UInt16):
        data = int(data)
    elif isinstance(data, dbus.Byte):
        data = int(data)
    elif isinstance(data, dbus.Double):
        data = float(data)
    elif isinstance(data, dbus.Array):
        data = [dbus_to_python(value) for value in data]
    elif isinstance(data, dbus.Dictionary):
        new_data = dict()
        new_key = ""
        for key in data.keys():
            new_key = dbus_to_python(key)
            new_data[new_key] = dbus_to_python(data[key])
        data = new_data
    return data

if __name__ == '__main__':
    
    global bus
   
    dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
    bus = dbus.SystemBus()

    bus.add_signal_receiver(property_changed, bus_name="org.bluez",
            dbus_interface="org.freedesktop.DBus.Properties",
            signal_name="PropertiesChanged",
            path_keyword="path")
    
    bus.add_signal_receiver(interfaces_added, bus_name="org.bluez",
            dbus_interface="org.freedesktop.DBus.ObjectManager",
            signal_name="InterfacesAdded")

    bus.add_signal_receiver(interfaces_removed, bus_name="org.bluez",
            dbus_interface="org.freedesktop.DBus.ObjectManager",
            signal_name="InterfacesRemoved")
    
    
    ## set up control-c handler
    signal.signal(signal.SIGINT, signal_handler)
    
    mainloop = GLib.MainLoop()
    #mainloop = GObject.MainLoop()

    ## get adapter
    adapter = find_adapter()
    try:
        ## start discovery
        adapter.StartDiscovery()
    except Exception as e:
        log_msg.error(e)
        sys.exit(1)

    mainloop.run()
    

Answer 1

The problem was narrowed down to a wrong value of Advertising Flag in the BLE Peripheral firmware. Though the flag was initialised to the right value ( 6 meaning BR/DR Not supported, General Discoverable) , this flag was getting reset to 0 as result of a bug in the firmware. Every subsequent advertising update had a zero value Advertising Flag .

Once the firmware bug was fixed the Raspberry PI started to discover and add the BLE peripheral to org.bluez

How did you arrive at the fact that advertising flags could be a problem? The observation that hctool and btmon were discovering the BLE device but bluetoothctl was unable to discover the device was puzzling. This led me to check bluetoothd debug messages from adapter.c and device.c.

The logs indicated that bluetoohd discovers the BLE device. Inspecting the code in device.c indicated that bluetoothd failed to add the device to org.blue tree as advertising flags value was 0. This drew our attention to Firmware and rectify the bug relating the Advertising Flag

How did Linux recognise and add the BLE Device if 0 value Advertising Flags were a problem? I don’t have a good answer as I could not reproduce this situation.

Here is one possibility of how this could have happened.

When the python program on Linux started device discovery , the value of Advertising Flags must have been 6 (the correct value, the firmware did initialise the flag values correctly) and therefore got added as an object in the org.bluez tree.