CODEWITHSUNDEEP
azuresdkazure-cosmosdbazure-iot-hubesp-idf
Adrian Wreyford
Adrian Wreyford

Reputation: 660

Azure IoTHub Telemetry messages from ESP32S3 are received in IoTHub, not Routing to CosmosDB endpoint, using ESP-IDF and azure-iot-middleware-freertos

On an ESP32S3 using ESP-IDF (not Arduino), and the azure-iot-middleware-freertos, specifically the sample_azure_iot_pnp.c, the ESP32S3-Device is sending telemetry data to the IoTHub, and I can view the data arriving, using the Azure IoT Explorer.

I have set up a route and a custom endpoint, to route to a NoSQL CosmosDB. The messages are not routed, even though the displayed body in the Telemetry pane of the IoTExplorer is correct.

If I cut and paste this body's content to the Test Routes Body, then the route tests OK with the Query condition I have set.

I suspect it has to do with setting the ContentEncoding = "utf-8", ContentType = "application/json". Also making sure the message is encoded in utf-8.

The example code uses the following function to setup the telemetry data:

uint32_t ulCreateTelemetry( uint8_t * pucTelemetryData,
                            uint32_t ulTelemetryDataSize,
                            uint32_t * ulTelemetryDataLength )
{
    int result = snprintf( ( char * ) pucTelemetryData, ulTelemetryDataSize,
                            myTELEMETRY_MESSAGE, myACTION); //sampleazureiotMESSAGE, xDeviceCurrentTemperature

    if( ( result >= 0 ) && ( result < ulTelemetryDataSize ) )
    {
        *ulTelemetryDataLength = result;
        result = 0;
    }
    else
    {
        result = 1;
    }

    return result;
}

The following is the calling function in a RTOS Task loop:

/* Publish messages with QoS1, send and process Keep alive messages. */
        for( ; ; )
        {
            /* Hook for sending Telemetry */
            if( ( ulCreateTelemetry( ucScratchBuffer, sizeof( ucScratchBuffer ), &ulScratchBufferLength ) == 0 ) &&
                ( ulScratchBufferLength > 0 ) )
            {
                xResult = AzureIoTHubClient_SendTelemetry( &xAzureIoTHubClient,
                                                           ucScratchBuffer, ulScratchBufferLength,
                                                           NULL, eAzureIoTHubMessageQoS1, NULL );
                configASSERT( xResult == eAzureIoTSuccess );
            }

            /* Hook for sending update to reported properties */
            ulReportedPropertiesUpdateLength = ulCreateReportedPropertiesUpdate( ucReportedPropertiesUpdate, sizeof( ucReportedPropertiesUpdate ) );

            if( ulReportedPropertiesUpdateLength > 0 )
            {
                xResult = AzureIoTHubClient_SendPropertiesReported( &xAzureIoTHubClient, ucReportedPropertiesUpdate, ulReportedPropertiesUpdateLength, NULL );
                configASSERT( xResult == eAzureIoTSuccess );
            }

            LogInfo( ( "Attempt to receive publish message from IoT Hub.\r\n" ) );
            xResult = AzureIoTHubClient_ProcessLoop( &xAzureIoTHubClient,
                                                     sampleazureiotPROCESS_LOOP_TIMEOUT_MS );
            configASSERT( xResult == eAzureIoTSuccess );

            /* Leave Connection Idle for some time. */
            LogInfo( ( "Keeping Connection Idle...\r\n\r\n" ) );
            vTaskDelay( sampleazureiotDELAY_BETWEEN_PUBLISHES_TICKS );
        }

The Telemetry Data is defined as follows:

#define myTELEMETRY_MESSAGE "{\"action\":\"%s\",\"ph_weight_device_mac\":\"XXXXXXXXXXXX\",\"ph_weight_index\":99,\"ph_weight_dev_nr\":0,\"ph_weight_block\":\"TEST\",\"ph_weight_activity\":\"ACTIVITY 2\",\"ph_weight_mode\":105,\"ph_weight_tag\":\"6D2517EBB4\",\"ph_weight_weight\":2.6202,\"ph_weight_date\":\"2022-12-07\",\"ph_weight_time\":\"13:45:12\"}"

The myACTION as follows:

#define myACTION "addPackhouseWeight"

The Azure IoT function that actually publishes the message:

AzureIoTResult_t AzureIoTHubClient_SendTelemetry( AzureIoTHubClient_t * pxAzureIoTHubClient,
                                                  const uint8_t * pucTelemetryData,
                                                  uint32_t ulTelemetryDataLength,
                                                  AzureIoTMessageProperties_t * pxProperties,
                                                  AzureIoTHubMessageQoS_t xQOS,
                                                  uint16_t * pusTelemetryPacketID )
{
    AzureIoTMQTTResult_t xMQTTResult;
    AzureIoTResult_t xResult;
    AzureIoTMQTTPublishInfo_t xMQTTPublishInfo = { 0 };
    uint16_t usPublishPacketIdentifier = 0;
    size_t xTelemetryTopicLength;
    az_result xCoreResult;

    if( pxAzureIoTHubClient == NULL )
    {
        AZLogError( ( "AzureIoTHubClient_SendTelemetry failed: invalid argument" ) );
        xResult = eAzureIoTErrorInvalidArgument;
    }
    else if( az_result_failed(
                 xCoreResult = az_iot_hub_client_telemetry_get_publish_topic( &pxAzureIoTHubClient->_internal.xAzureIoTHubClientCore,
                                                                              ( pxProperties != NULL ) ? &pxProperties->_internal.xProperties : NULL,
                                                                              ( char * ) pxAzureIoTHubClient->_internal.pucWorkingBuffer,
                                                                              pxAzureIoTHubClient->_internal.ulWorkingBufferLength,
                                                                              &xTelemetryTopicLength ) ) )
    {
        AZLogError( ( "Failed to get telemetry topic: core error=0x%08lx", xCoreResult ) );
        xResult = AzureIoT_TranslateCoreError( xCoreResult );
    }
    else
    {
        xMQTTPublishInfo.xQOS = xQOS == eAzureIoTHubMessageQoS1 ? eAzureIoTMQTTQoS1 : eAzureIoTMQTTQoS0;
        xMQTTPublishInfo.pcTopicName = pxAzureIoTHubClient->_internal.pucWorkingBuffer;
        xMQTTPublishInfo.usTopicNameLength = ( uint16_t ) xTelemetryTopicLength;
        xMQTTPublishInfo.pvPayload = ( const void * ) pucTelemetryData;
        xMQTTPublishInfo.xPayloadLength = ulTelemetryDataLength;

        /* Get a unique packet id. Not used if QOS is 0 */
        if( xQOS == eAzureIoTHubMessageQoS1 )
        {
            usPublishPacketIdentifier = AzureIoTMQTT_GetPacketId( &( pxAzureIoTHubClient->_internal.xMQTTContext ) );
        }

        /* Send PUBLISH packet. */
        if( ( xMQTTResult = AzureIoTMQTT_Publish( &( pxAzureIoTHubClient->_internal.xMQTTContext ),
                                                  &xMQTTPublishInfo, usPublishPacketIdentifier ) ) != eAzureIoTMQTTSuccess )
        {
            AZLogError( ( "Failed to publish telemetry: MQTT error=0x%08x", xMQTTResult ) );
            xResult = eAzureIoTErrorPublishFailed;
        }
        else
        {
            if( ( xQOS == eAzureIoTHubMessageQoS1 ) && ( pusTelemetryPacketID != NULL ) )
            {
                *pusTelemetryPacketID = usPublishPacketIdentifier;
            }

            AZLogInfo( ( "Successfully sent telemetry message" ) );
            xResult = eAzureIoTSuccess;
        }
    }

    return xResult;
}

The data shows up in the Telemetry Pane of Azure IoT Explorer:

Fri Dec 09 2022 12:35:24 GMT+0200 (South Africa Standard Time):
{
  "body": {
    "action": "addPackhouseWeight",
    "ph_weight_device_mac": "XXXXXXXXXXXX",
    "ph_weight_index": 99,
    "ph_weight_dev_nr": 0,
    "ph_weight_block": "TEST",
    "ph_weight_activity": "ACTIVITY 2",
    "ph_weight_mode": 105,
    "ph_weight_tag": "AD2512EBB4",
    "ph_weight_weight": 2.6202,
    "ph_weight_date": "2022-12-07",
    "ph_weight_time": "13:45:12"
  },
  "enqueuedTime": "Fri Dec 09 2022 12:35:24 GMT+0200 (South Africa Standard Time)",
  "systemProperties": {
    "iothub-connection-device-id": "XXXXXXXXXXXX",
    "iothub-connection-auth-method": "{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
    "iothub-connection-auth-generation-id": "638052575116976851",
    "iothub-enqueuedtime": 1670582124804,
    "iothub-message-source": "Telemetry",
    "dt-dataschema": "dtmi:com:loadassist:Packhouse;2"
  }
}

So, it is arriving, but probably Base64 encoded and Azure IoT Explorer just deals with it and displays properly.

But I suspect, that the routing is not working, as the example code does not set the encoding and the type of data as explained above. There are Arduino examples where they setup and encode a message, but nothing to that effect in this SDK.

I cannot see if they do it anywhere else in their example code.

I don't know how to go about this with the SDK, and the used MQTT WebSocket.

I would also like to set a property value, so that I can possibly query and route on that, and not have the "action": "addPackhousWeight" stored with the weight data to cosmosDB.

Well I hope I have been clear enough. I tried a support Ticket with Microsoft, but it was closed by them and unanswered. Thx

Comment Added: I found the following in the sample_azure_iot.c not included in the sample_azure_iot_pnp.c :

/* Create a bag of properties for the telemetry */ 
        xResult = AzureIoTMessage_PropertiesInit( &xPropertyBag, ucPropertyBuffer, 0, sizeof( ucPropertyBuffer ) );
        configASSERT( xResult == eAzureIoTSuccess );

        xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "content-encoding", sizeof( "content-encoding" ) - 1,
                                                    ( uint8_t * ) "utf-8", sizeof( "utf-8" ) - 1 );
        configASSERT( xResult == eAzureIoTSuccess );


xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "content-type", sizeof( "content-type" ) - 1,
                                                            ( uint8_t * ) "application%2Fjson", sizeof( "application%2Fjson" ) - 1 );
                configASSERT( xResult == eAzureIoTSuccess );

or as follows:

/* Create a bag of properties for the telemetry */ 
        xResult = AzureIoTMessage_PropertiesInit( &xPropertyBag, ucPropertyBuffer, 0, sizeof( ucPropertyBuffer ) );
        configASSERT( xResult == eAzureIoTSuccess );

        xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "contentEncoding", sizeof( "contentEncoding" ) - 1,
                                                    ( uint8_t * ) "utf-8", sizeof( "utf-8" ) - 1 );
        configASSERT( xResult == eAzureIoTSuccess );

xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "contentType", sizeof( "contentType" ) - 1,
                                                            ( uint8_t * ) "application%2Fjson", sizeof( "application%2Fjson" ) - 1 );
                configASSERT( xResult == eAzureIoTSuccess );

Tried both content-type, and contentType etc. But alas, still not routing ...

Upvotes: 0

Views: 471

Answers (1)

Adrian Wreyford
Adrian Wreyford

Reputation: 660

At last I got it to work:

xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "$.ce", sizeof( "$.ce" ) - 1,
                                                    ( uint8_t * ) "utf-8", sizeof( "utf-8" ) - 1 );
        configASSERT( xResult == eAzureIoTSuccess );

xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "$.ct", sizeof( "$.ct" ) - 1,
                                                            ( uint8_t * ) "application%2Fjson", sizeof( "application%2Fjson" ) - 1 );
                configASSERT( xResult == eAzureIoTSuccess );

I found the solution here: https://azure.microsoft.com/sv-se/blog/iot-hub-message-routing-now-with-routing-on-message-body/

enter image description here

So adding "$.ce" and "$.ct" helped to get it to work. Literally took me days. Hope it will help someone else.

I also learnt this:

        /**
         *   @note The properties init API will not encode properties. In order to support
         *       the following characters, they must be percent-encoded (RFC3986) as follows:
         *         - `/` : `%2F`
         *         - `%` : `%25`
         *         - `#` : `%23`
         *         - `&` : `%26`
         *       Only these characters would have to be encoded. If you would like to avoid the need to
         *       encode the names/values, avoid using these characters in names and values.
         */

Upvotes: 0

Related Questions