CODEWITHSUNDEEP
azure-databricksdatabricks-notebook
Mohammed Arif
Mohammed Arif

Reputation: 61

Get Command number in databricks

How can i get the current command number dynamically from a cell in Azure databricks notebook? Is there any dbutils command to retrieve it?

For below cell, it should return 1.

Databricks command example

Upvotes: 0

Views: 149

Answers (1)

Dileep Raj Narayan Thumula
Dileep Raj Narayan Thumula

Reputation: 3250

There is no direct built-in function in dbutils or Spark to retrieve the current command number dynamically from a cell in an Azure Databricks notebook.

I have tried the following approach:

spark._jsparkSession.sparkContext().getLocalProperty("spark.databricks.notebook.id")
result = spark.sql("SELECT * FROM default.my_table")
print(f"Current command number: {command_number}")

enter image description here

Result:

Current command number: 2585995787002146

spark._jsparkSession.sparkContext().getLocalProperty("spark.databricks.notebook.id"): The command above retrieves the number of the current cell in the Databricks notebook.

  • It uses the SparkSession (spark) to access the underlying Java SparkSession, then gets the local property "spark.databricks.notebook.id" which holds the command number.

  • The code uses the Spark context (spark._jsparkSession.sparkContext()) to access a property that represents the current command number associated with the notebook.

  • This property, spark.databricks.notebook.id, uniquely identifies each command execution within the notebook. The getLocalProperty method is then used to retrieve the value of this property, which corresponds to the command number.

Upvotes: 1

Related Questions