CODEWITHSUNDEEP
azure-data-explorer
der.Schtefan
der.Schtefan

Reputation: 97

ADX: Preventing blob metadata to interfere with ingestion?

We want to setup data ingestion whenenever a new blob is uploaded into our Storage Account (via Event Grid > Event Hub route). The following page explains that metadata set on the blob can interfere with the ingestion: https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-event-grid-overview

3rd parties will upload BLOBs as part of external data ingest with an SAS token to their particular directory, where it would trigger an ingest.

What worries us if anybody messes with the kustoTable, kustoCreationTime, or kustoExtentTags metadata properties when uploading their daily blob, causing all sorts of issues.

Can the honoring of these metadata properties on the blobs be deactivated or the problem somehow mitigated?

Upvotes: 1

Views: 179

Answers (1)

Vladik Branevich
Vladik Branevich

Reputation: 1175

The core requirement that led us to implement the handling of blob metadata properties was just the opposite - the desire to manage a single EG data connection that is able to "feed" multiple tables.

And hence the blob metadata has higher precedence.

In your case what you can do is create an Azure Function that will forcefully remove all undesired metadata properties from the blobs before renaming or copying them so that they are picked up by the EG.

Or if you go that way and put a function as the EG notification sink, your function can simply use our ingest SDK to submit the blob for ingestion using queued ingestion.

Upvotes: 0

Related Questions