CODEWITHSUNDEEP
apache-sparkhadoophive
SuperAadi
SuperAadi

Reputation: 737

What is the use of winutils.exe?

I am running apache spark on windows(locally) using intellij.

I chose enableHiveSupport while creating spark session object.

I converted a dataframe into temp view and ran some queries.

Initially I got an error that tmp/hive does not exist. So I created one on the C: drive.

Then I got an error that tmp/hive is not writable.

So I changed the permissions in the file properties. But I still got the same error.

After researching I found the solution i.e use winutils.exe to change the permissions.

So what exactly is winutils.exe? Where is it used spark? the tmp/hive/username was empty after I ran the application.

Thank you

Upvotes: 1

Views: 1568

Answers (1)

Ged
Ged

Reputation: 18108

I advise you run on linux, but if using Windows for Spark accessing Hadoop on Windows, then cmd> winutils.exe chmod -R 777 D:\tmp\hive allows you to read and write to this pseudo Hadoop.

Upvotes: 1

Related Questions