CODEWITHSUNDEEP
powershellazurepysparkazure-hdinsight
YuGagarin
YuGagarin

Reputation: 351

How to submit a Spark job on HDInsight via Powershell?

Is there a way to submit a Spark job on HDInsight via Powershell? I know it can be done via activity in Azure Data Factory, but is there a way to submit python script to pyspark HDInsight from Powershell cmdlet?

Upvotes: 0

Views: 731

Answers (1)

Shui shengbao
Shui shengbao

Reputation: 19195

Based on my knowledge, there is no Azure PowerShell command could do this.

You could use Apache Spark REST API, which is used to submit remote jobs to an Azure HDInsight Spark cluster. Please refer to this feedback.

HDInsight allows remote job submission through the REST API using Livy. It is part of the recent Spark release on Linux. https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-livy-rest-interface/

Upvotes: 1

Related Questions