ML Spark job in Azure databricks

Question

Currently we are using ML pyspark jobs in Azure HDInsight cluster(version: HDI 3.6). Is it possible to use the same pyspark jobs in Azure Databricks without making much changes?

Thanks in advance!

Answer 1

I would say yes, but take in consideration the following things :

• Make sure that the Python and Spark version of your HDInsight cluster are similar or compatible with the Databricks runtime choosen for the cluster.
• Are you using any external library? If so import it to your workspace and attach it to the cluster.
• Which data source are you using for your jobs? You may have to mount the data source into your databricks workspace, for example to mount a blob storage ( https://docs.databricks.com/spark/latest/data-sources/azure/azure-storage.html#mount-azure-blob-storage-containers-with-dbfs )

I hope this info helps you

Regards,

Juan