CODEWITHSUNDEEP
azuredatabricksazure-databrickskqlazure-log-analytics
Shaan
Shaan

Reputation: 1

Azure DataBricks - Looking to query "workflows" related logs in Log Analytics (ie Name, CreatedBy, RecentRuns, Status, StartTime, Job)

We're looking to fetch logs of the section "workflows" from Azure DataBricks to our Log Analytics Workspace. We have our log analytics workspace connected with all logs enabled in diagnostic settings. Inside Workflows, were are seeing an output table with the data we need but can't access in Log Analytics to display on a workbook.. I've tried to query using DatabricksJobs operation, but it does not output the below columns we see in the workflow output table?

Start time Job Run as Launched Duration Status Run parameters

We have tried to query using all the available operations when enabling every diagnostic parameter. We were hoping they're would be a parameter to query Workflow related jobs and output a similar table were seeing in databricks in our log analytics workspace.

Upvotes: 0

Views: 416

Answers (1)

JayashankarGS
JayashankarGS

Reputation: 8020

By default the logs for databricks will be having below schema.

enter image description here

If you want the data which is in runs tab

  1. Start time
  2. Run ID
  3. Launched
  4. Duration
  5. Status
  6. Error code
  7. Run parameters

You need to create custom table with above schema and log the details.

Using this api you can get details and extract he required data you need.

After getting the result you send it to log analytics using python code. Refer this stack solution on how to send data to log analytics table.

Upvotes: 0

Related Questions