CODEWITHSUNDEEP
azure-eventhubazure-data-exploreradx
suresh pasupunuri
suresh pasupunuri

Reputation: 1

How to send Json & Avro format events to Azure Event Hub and finally Ingesting them into Azure Data Explorer Table. Using .Net (c#)

My requirement is using Azure Event Hub to capture events in format Json and/or Avro format and finally ingesting those events into ADX. Also don't want to capture these events in to Blob storage etc and using Event Grid option on top of it. Want to capture these events directly into ADX.

After going through several documentation and online sites on these topics, below are the high-level steps as per my understanding, however was not able to complete this due to several issues, so reaching out for help to see if any blog /video which shows this process end to end with a simple example ?

  1. Create Event Hub Name space.
  2. Create a EventHub (topic)
  3. Under Event Hub Name space, under "Schema Registry" create a Schema group (type : Json/Avro) and schema. (Not enough details found on how to define json schema)
  4. Using .Net SDK (c#) we can validate the event and send and receive events. reference link: https://learn.microsoft.com/en-us/azure/event-hubs/schema-registry-dotnet-send-receive-quickstart
  5. Create one-click ingestion (Event Hub connection) in ADX database. For this step, tried multiple options like creating a brand new table with one click ingestion and pre-creating table and mapping, but none of the options worked and was not able to capture events into the table. This is where really need more details on how to configure table/mapping and how to create an event hub connection etc. How to process Avro and JSON format events exactly.

Here is the KQL script for the pre-creating table and mapping I am using.

.create table test123 (
 RGUID: string,
 EnvironmentName: string, 
 MachineName: string,
 DateTimeUTC: long
)


.create table test123 ingestion avro mapping 'test123Mapping1' '[{"column":"RGUID","Properties":{"Field":"$.RGUID"}},{"column":"EnvironmentName","Properties":{"Field":"$.EnvironmentName"}},{"column":"MachineName","Properties":{"Field":"$.MachineName"}},{"column":"DateTimeUTC","Properties":{"Field":"$.DateTimeUTC"}}]'

Thanks

Edit-1: There are 2 issues mainly while creating DB connection from Event Hub.

  1. Created DB connection at DB level as per the screenshot below. The connection created successfully, but events are getting dropped with Error Code BadRequest_InvalidBlob

EventHub DB Connection

  1. We can also create DB connection at the Table level. Right-click on the table and select "Ingest Data" and here I don't see "AVRO" format is available.

Table Level Ingest Data wizard

Upvotes: 0

Views: 1207

Answers (2)

Vladik Branevich
Vladik Branevich

Reputation: 1175

  1. ADX does not support EH Schema registry, the schemas must be embedded with the data.
  2. If you are trying to ingest both JSON and AVRO data on the same data connection, you would need to specify format and ingestion mapping on every event, as explained here: https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-event-hub-overview#ingestion-properties

Upvotes: 0

Ariel Yehezkely
Ariel Yehezkely

Reputation: 91

If you already know how to create the table and the ingestion mapping, you can create the Data Connection directly from the Azure portal instead of going through the one-click wizard:

Create an Event hub data connection

Upvotes: 0

Related Questions