CODEWITHSUNDEEP
azureazure-hdinsight
Megamini
Megamini

Reputation: 313

Installing external python package on HDInsight with Azure CLI : RdfeResourceHandlerException

I want to install the pyobdc python package on a MS Azure HDInsight cluster, to use in a pyspark job. Following this, I trying to get this done with a "script action". The bash script is :

#!/usr/bin/env bash
sudo /usr/bin/anaconda/bin/conda install pyodbc

Among options, the script action can be submitted using the web interface (azure portal) or a CLI command line (doc). In the web interface, the submit option is deactivated with the info "New script actions can be submitted after the current cluster operation finishes.". I have no idea what this cluster operation is.

I tried to used the CLI command line:

$ azure hdinsight script-action create my_cluster_name -g my_resource_group -n ipyodbc -u adl://my_data_lake_store.azuredatalakestore.net/clusters/scripts/script_actions/install_pyodbc.sh -t headnode;workernode

which fail with:

info: Executing command hdinsight script-action create /error:
Exception of type 'Microsoft.ClusterServices.RDFEProvider.ResourceTypes.Models.RdfeResourceHandlerException' was thrown. -error: Error information has been recorded to /home/myself/.azure/azure.err error: hdinsight script-action create command failed

I tried to change the script location with a public uri (dropbox), I got the same error. Any help or workaround would be much appreciated ! Cheers !

Upvotes: 1

Views: 817

Answers (1)

James Burke
James Burke

Reputation: 2387

I was also getting the ambiguous error 'Microsoft.ClusterServices.RDFEProvider.ResourceTypes.Models.RdfeResourceHandlerException'.

In my case it was due to not including the path to conda in the command and simply using conda instead of /usr/bin/anaconda/bin/conda.

Two things you might try to resolve your issue:

  • remove sudo, you shouldn't need this for executing the command on the custer and potentially prompts for a password.
  • add the -y switch to the command e.g. /usr/bin/anaconda/bin/conda install -y pyodbc this prevents prompts

My assumption is that the hdinsight script-action create command doesn't like being held up and if it gets stuck it returns this uninformative and from what I could find (or couldn't) undocumented error.

Upvotes: 0

Related Questions