CODEWITHSUNDEEP
azure-data-explorerkql
Fonzie Ghaffar
Fonzie Ghaffar

Reputation: 1

Optimization for Kusto query

How to optimize the KQL parsing strings to get quick output, less than 10 seconds;

Example: where message contains "this is the error code following this format –"

should we change it to hex to see if this parses quickly : where message contains "7468697320697320746865206572726f7220636f646520666f6c6c6f77696e67207468697320666f726d61742096"``

Upvotes: 0

Views: 759

Answers (1)

David דודו Markovitz
David דודו Markovitz

Reputation: 44991

From the documentation (String Operators):

"Kusto builds a term index consisting of all terms that are three characters or more, and this index is used by string operators such as has,!has, and so on. If the query looks for a term that is smaller than three characters, or uses a contains operator, then the query will revert to scanning the values in the column. Scanning is much slower than looking up the term in the term index."

Instead of contains use one of the has[...] operators described in the documentation, preferably a case-sensitive one (has_cs, hasprefix_cs, hassuffix_cs etc.).

Search for fewer terms as possible, e.g., if ... has_cs "ErrorCode" is enough to filter out the records you need, prefer it over searches like ... has "this is the ErrorCode following this format"

P.S.
I've been doing similar searches, over milliards of records, on my own development ADX cluster, in sub-seconds.

Upvotes: 3

Related Questions