CODEWITHSUNDEEP
apache-sparkspark-structured-streamingazure-databricks
dag
dag

Reputation: 290

automated cluster AND high-concurrency Databricks cluster

Databricks allows to create two flavors for clusters: High-Concurrency and Standard. The High-Concurrency does not accept Scala code... :- (. The choice is visible for the interactive-cluster but not for automated-cluster.

My concern is about one streaming application where one job is submitted for executing iot acquisition. So it will be a job , the question is about the cluster : will it be a high-concurrency or a standard one ?

Thank you!

Upvotes: 1

Views: 774

Answers (1)

Daniel
Daniel

Reputation: 1242

That depends if you use new automated cluster in job definition or select to use existing cluster. Automated clusters are considered data engineering loads and do not support high-concurrency (which wouldn't make any sense anyway as job clusters are created for each of the jobs).

Upvotes: 1

Related Questions