CODEWITHSUNDEEP
pysparkazure-databricks
S.J.
S.J.

Reputation: 31

Can pure python script (not pyspark) run in parallel in a cluster in Azure Databricks?

I want to migrate my python scripts from local to run on cloud, specifically on a cluster created on Azure Databricks.

  1. Can pure python script run in parallel (using multiple nodes in a cluster at the same time) without having to be converted into pyspark?
  2. Is it possible to check whether the job is running in parallel?

Upvotes: 2

Views: 824

Answers (1)

srikanth holur
srikanth holur

Reputation: 780

  1. No
  2. If you have submitted a spark job, you can see logs and it will show you number of executors it is using. That's how you 'll know job is running in parallel

Upvotes: 1

Related Questions