CODEWITHSUNDEEP
databricksdatabricks-asset-bundle
moro clash
moro clash

Reputation: 319

Databricks assets bundles deploying on DB workflow

the project is wrapped as DAB, and all workflow runs as I set, the only problem is that I have a personal cluster with different specs, but sometimes the workflow uses my cluster instead of creating a new one with specs as exist in YAML, in YAML there's no specific cluster_id there, sometimes it creates a new cluster as exist in YAML and sometimes it uses the personal cluster.

What are the possible areas I should check?

Upvotes: 0

Views: 175

Answers (1)

Deva
Deva

Reputation: 66

It’s tough to assist without more details, but here’s where DAB typically pulls cluster configurations from:

  1. First and foremost, check the job’s YAML file. Hope you have things sorted here.
  2. If you’re using the extension, double-check the selected cluster in the UI and the “Override Job Clusters in Bundle” checkbox.
  3. Look for cluster_id in your .databrickscfg file, in case you used Databricks CLI to configure.
  4. project.json file under .databricks folder will be holding the project level configuration for cluster.
  5. And databricks's own terraform files - .databricks/local/terraform/bundle.tf.json or terraform.tfstate - Remember, DAB is essentially a wrapper around Terraform, so these files likely hold the keys to resolving your issue.

Upvotes: 0

Related Questions