CODEWITHSUNDEEP
hadoophdfsazure-blob-storageapache-nifi
user9238406
user9238406

Reputation: 11

Connecting to HDFS that uses Azure Blob as Storage in Apache Nifi

I am trying to connect to HDFS and add files that uses Azure Blob as Storage. In Apache Nifi I have configured the HDFS connection using hdfs-site.xml and core-site.xml files.

The PutHDFS component in Apache Nifi throws the following error.

10:01:39 UTC ERROR
PutHDFS. Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found ↳ causes: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found
 
10:01:39 UTC ERROR
PutHDFS HDFS Configuration error - java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found ↳ causes: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found
 
10:01:39 UTC ERROR
PutHDFS org.apache.hadoop.fs.azure.NativeAzureFileSystem: java.lang.ClassNotFoundException: org.apache.hadoop.fs.azure.NativeAzureFileSystem

Are there additional properties that have to be added while establishing the HDFS connection?

If there are other ways of connecting to HDFS that uses Azure Blob Storage will be welcomed.

Upvotes: 1

Views: 260

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191681

Seems like your Nifi JVM classpath does not have hadoop-azure JAR files.

Pick the version that matches your hadoop installation - https://search.maven.org/artifact/org.apache.hadoop/hadoop-azure

Alternatively, don't use PutHDFS; use PutAzureBlobStorage

Upvotes: 1

Related Questions