CODEWITHSUNDEEP
azurepysparkdatabricksazure-databricksdatabricks-connect
batman_special
batman_special

Reputation: 125

add column to existing dataframe from widgets values using pyspark

I have a dataframe where i need to add a column from the widget value that is being passed. I am trying the below code but its not helping in anyways. When we display(pdf) we should also see the ID column has also been added.

  pdf=pTF.withColumn('ID',lit(dbutils.widgets.text("ID", "eiheifhj", label="pro_ID")).cast(StringType()))

The above code is puhsing all nulls into the columns ID. Could you please help. I would request the experts to help in the above challenge

Upvotes: 0

Views: 824

Answers (1)

Anand Vidvat
Anand Vidvat

Reputation: 1058

In the following code, you are actually defining the widget instead of fetching the value the widget contains.

dbutils.widgets.text("ID", "eiheifhj", label="pro_ID")
  • "ID" is the name of your widget.
  • "eiheifhj" is the default value of the widget.
  • "pro_ID" is the widget label in UI

so if you want to fetch the value of the 'ID' widget, you can use the following method.

dbutils.widgets.get(<widget_name>)

hence your program should be modified as follows :

pdf=pTF.withColumn('ID',lit(dbutils.widgets.get("ID")).cast(StringType()))

Upvotes: 1

Related Questions