CODEWITHSUNDEEP
azure-eventhubazure-data-explorerdata-ingestionadxeventhub
Bhavna Devgude
Bhavna Devgude

Reputation: 11

Guidance on EventHub data ingestion process

I am sending the events from the application to eventHub.I am able to monitoring the health graph of eventHub due to limited access. there I have noticed that Metric Events received and Events processes count but on examining the configured primary table for the particular eventHub no recent data was ingested.

Currently I have been suggested to enable the capture feature for the eventHub and configure storage account which has few of logs categories enabled. Once that capture feature is got enabled 1. What do I need to check reagrding the data ingestion failure? 2. If data ingestion is ongoing, what steps can I take to ensure successful ingestion into the configured primary table?

  1. cluster01> diagnostic settings > Avro> capture ON >select logs categories > archived to storage account > specify the storage account Name > SAVE. these are steps we are following as of now so, I have a question that, does it affect only that eventHub or all eventHubs available in that cluster?

Looking forward to the suggestions.

Upvotes: 1

Views: 221

Answers (1)

Sampath
Sampath

Reputation: 3639

The .show ingestion failures command return all recorded ingestion failures, providing details such as operation ID, database, table, failed timestamp, ingestion source path, failure details, failure kind, root activity ID, operation kind, error code, principal, user, ingestion properties, and number of sources.

.show ingestion failures

You can also use filters to narrow down the results. For instance, to retrieve ingestion failures for a specific operation ID, you can use:

.show ingestion failures with (OperationId = '3827def6-0773-4f2a-859e-c02cf395deaf')

This the reference link for .show ingestion failures command in Azure Data Explorer.

  • When sending messages in event Hub , you include custom properties in the message properties section.

these custom properties to be ingested into Azure Data Explorer, you need to embed them into the body section of the event data object.

The following example compares the events data object containing custom property customProperty as defined by Event Hubs (Top) with the embedded property required for ingestion (down).

{
"body":{
"value": 42
},
"properties":{
"customProperty": "123456789"
}
}


{
"body":{
"value": 42,
"customProperty": "123456789"
}
}
  • For more details refer to this link for Ingest from event hub - Azure Data Explorer

enter image description here Follow this for this steps in so to create table

  • Refer to this link for Continuous data export Azure Data Explorer

Refer to this so to get data from storage to Azure Data Explorer tables .

  • You followed this process below cluster01> diagnostic settings > Avro> capture ON >select logs categories > archived to storage account > specify the storage account Name > SAVE. It only track the all event in the Azure Data Explorer cluster only and send to storage. It don't effect the event Hub which are connected to data source/ added to the database.

Upvotes: 1

Related Questions