CODEWITHSUNDEEP
databricksazure-databricks
Stefan Papp
Stefan Papp

Reputation: 2255

Passing Context information to task in DataBricks Workflows

I need in my task the scheduling date of the job run to run queries based on a timestamp. Is there a way to do it with Parameters?

Adding parameters to task

I am able to set the start-time (https://learn.microsoft.com/en-us/azure/databricks/data-engineering/jobs/jobs#task-parameter-variables). But I need the exact scheduling time.

Upvotes: 1

Views: 3330

Answers (2)

AngryCoder
AngryCoder

Reputation: 496

I guess that you are asking this because your job has several tasks. If that's the case, you can use Dbutils.jobs.taskvalues for doing so, in the first task of your job, you can get the current timestamp as the first command executed and then set a job taskValue:

from datetime import datetime

utc_timestamp = datetime.utcnow()
dbutils.jobs.taskValues.set("job_start_timestamp", utc_timestamp)

And in the other task in which you want to make use of these values:

job_start_timestamp = dbutils.jobs.taskValues.get(taskKey="first_task_name",key="job_start_timestamp")

Upvotes: 3

Ganesh Chandrasekaran
Ganesh Chandrasekaran

Reputation: 1936

The schedule date/time is set at the job or workflow level. Not at the task level.

Upvotes: 0

Related Questions