CODEWITHSUNDEEP
azureazure-eventhubazure-data-explorer
PSKP
PSKP

Reputation: 1375

Event hub & Azure Data Explorer not ingesting all messages

For the last 12 hours, I am noticing Outgoing Messages in the Event hub are too few than incoming messages. & more worrying thing is that event hub Outgoing Messages frequency is the same (Like service is intentionally doing it). You can see in the below images that after 12 PM Outgoing Messages are 100 per half hour. But there are more messages available.

enter image description here

Only Outgoing messages with Zoom

enter image description here

What I am doing

We are pushing application logs to Event Hub & Developer can watch these logs in ADX cluster.

Event hub configuration

ADX configuration

Event hub Ingestion metrics in ADX

enter image description here

About batching

After some googling, I found about batching. I thought may be resources are misconfigured but this is not the case

DB batch policy

.show database altslogsdb policy ingestionbatching

Output

"PolicyName": IngestionBatchingPolicy,
"EntityName": [altslogsdb],
"Policy": null,
"ChildEntities": [
  "alts_logs_table"
],
"EntityType": Database,

Table batch policy

.show table alts_logs_table policy ingestionbatching

Output

"PolicyName": IngestionBatchingPolicy,
"EntityName": [altslogsdb].[alts_logs_table],
"Policy": null,
"ChildEntities": ,
"EntityType": Table,

May be issue got resolved automatically

After more than 24 hours, it may be got automatically resolved. I also created a support ticket. I don't know it got resolved automatically or Microsoft resolved it. Waiting for a mail from a support person.

enter image description here

As you can see, 3 spikes at last, I think that event hub was dropping messages because no messages are available in ADX & issue is there in that period also.

Update

Created ticket for Microsoft. Event hub and Azure data explorer teams are looking into it. The ticket is open for the last 2 months.

Upvotes: 0

Views: 1278

Answers (1)

Avnera
Avnera

Reputation: 7618

There are multiple checks that you should do:

  1. As Alex said above, take a look at the "insights" blade to see if there are ingestion failures on Azure Data Explorer side.
  2. You can check the EventHub ingestion metrics in your Azure Data Explorer cluster metrics: enter image description here
  3. Check to see if there are errors on the EventHub such as throtteling

If you can't find the information there, please open a support ticket.

Upvotes: 0

Related Questions