CODEWITHSUNDEEP
apache-sparkpysparkdatabricksdatabricks-workflowsdatabricks-dbx
Borislav Blagoev
Borislav Blagoev

Reputation: 197

How can I pass and than get the passed arguments in databricks job

I'm trying to pass and get arguments in my databricks job it's a spark_python_task type IT IS NOT A NOTEBOOK. I deployed my job with dbx from pycharm. I have deployment.json file where I configure deployment stuff.

Upvotes: 1

Views: 4144

Answers (1)

Alex Ott
Alex Ott

Reputation: 87154

If you follow documentation about deployment file, you can see that you can specify parameters as parameters array:

{
  "name": "this-parameter-is-required!",
  "spark_python_task": {
      "python_file": "path/to/entrypoint.py" 
  },
  "parameters": [
     "--conf-file",
     "conf/test/sample.json"
  ]
}

Parameters are passed as command-line parameters, so you can get them from the code just using the sys.argv, or built-in argparse library.

Upvotes: 2

Related Questions