build hadoop 2.2 on windows

Question

I am trying to install hadoop on windows, it is so troublesome so far.

here is error i keep getting

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Cannot run program "msbuild" (in directory "C:\hdfs\hadoop-common-project\hadoop-common"): CreateProcess error=2, The system cannot find the file specified -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

I have msbuild.exe, windows sdk7.1, cygwin64, maven3.0.5, jdk7 all set up in environment on my windows 7 x64 box.

This is hadoop 2.2.0 src which i downloaded from apache website.

under system environment, i have set

C:\Windows\Microsoft.NET\Framework64\v4.0.30319

this is the path to msbuild, but still doesn't work, does anyone know how to fix this?

thanks

update: I changed pom.xml at hadoop-common-project\hadoop-common a little from

<executable>msbuild</executable>

to

<executable>msbuild.exe</executable>

Now, I'm getting following error

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

Answer 1

Even after following all the steps mentioned in the document and here, I got the following error:

Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec 
(compile-ms-winutils) on project hadoop-common

The detailed error said:

You are attempting to build a Win32 application from an x64 environment. 
If using the Windows 7.1 SDK build environment, type setenv /x86

So I added the following environment variable and that resolved the issue

Platform: x64

Answer 2

You may need CMake.exe - get it using official link http://www.cmake.org/download/

Open file BUILDING.txt which located in unzipped hadoop-2.6.0-src.tar.gz or your version.
It say you how to build for Windows - check all of this:

My BUILDING.txt says:

Building on Windows
Requirements:

* Windows System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK or Visual Studio 2010 Professional
* Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279    

Answer 3

I have followed following steps to install Hadoop 2.2.0

Steps to build Hadoop bin distribution for Windows

1. Download and install Microsoft Windows SDK v7.1.

2. Download and install Unix command-line tool Cygwin.

3. Download and install Maven 3.1.1.

4. Download Protocol Buffers 2.5.0 and extract to a folder (say c:\protobuf).

5. Add Environment Variables JAVA_HOME, M2_HOME and Platform if not added already. Note : Variable name Platform is case sensitive. And value will be either x64 or Win32 for building on a 64-bit or 32-bit system. Edit Path Variable to add bin directory of Cygwin (say C:\cygwin64\bin), bin directory of Maven (say C:\maven\bin) and installation path of Protocol Buffers (say c:\protobuf).

6. Download hadoop-2.2.0-src.tar.gz and extract to a folder having short path (say c:\hdfs) to avoid runtime problem due to maximum path length limitation in Windows.

7. Select Start --> All Programs --> Microsoft Windows SDK v7.1 and open Windows SDK 7.1 Command Prompt. Change directory to Hadoop source code folder (c:\hdfs). Execute mvn package with options -Pdist,native-win -DskipTests -Dtar to create Windows binary tar distribution.

8. If everything goes well in the previous step, then native distribution hadoop-2.2.0.tar.gz will be created inside C:\hdfs\hadoop-dist\target\hadoop-2.2.0 directory.

Install Hadoop

1. Extract hadoop-2.2.0.tar.gz to a folder (say c:\hadoop).

2. Add Environment Variable HADOOP_HOME and edit Path Variable to add bin directory of HADOOP_HOME (say C:\hadoop\bin).

Configure Hadoop

C:\hadoop\etc\hadoop\core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:9000</value>
        </property>
</configuration>

C:\hadoop\etc\hadoop\hdfs-site.xml

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/hadoop/data/dfs/namenode</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/hadoop/data/dfs/datanode</value>
        </property>
</configuration>

C:\hadoop\etc\hadoop\mapred-site.xml

<configuration>
        <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
        </property>
</configuration>

C:\hadoop\etc\hadoop\ yarn-site.xml

<configuration>
        <property>
           <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
        </property>
        <property>
           <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
           <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
</configuration>

Format namenode

For the first time only, namenode needs to be formatted.

C:\Users\abhijitg>cd c:\hadoop\bin 
c:\hadoop\bin>hdfs namenode –format

Start HDFS (Namenode and Datanode)

C:\Users\abhijitg>cd c:\hadoop\sbin
c:\hadoop\sbin>start-dfs

Start MapReduce aka YARN (Resource Manager and Node Manager)

C:\Users\abhijitg>cd c:\hadoop\sbin
c:\hadoop\sbin>start-yarn
starting yarn daemons

Total four separate Command Prompt windows will be opened automatically to run Namenode, Datanode, Resource Manager, Node Manager

Reference : Build, Install, Configure and Run Apache Hadoop 2.2.0 in Microsoft Windows OS

Answer 4

I was able to fix this issue by below two steps.

1) Add the MsBuild path(C:\Windows\Microsoft.NET\Framework64\v4.0.30319) to Environment Variable Path

2) Installing Microsoft-Visual-Studio-2010-Professional http://download.cnet.com/Microsoft-Visual-Studio-2010-Professional/3000-2212_4-10618634.html

Note : Before installing Visual-Studio-2010-Professional I tried Windows-SDK-7.1 which did not help me. So I uninstalled Windows-SDK-7.1 before installing Visual-Studio-2010-Professional. Also the link I provided for Visual-Studio-2010-Professional is a trial version.

Answer 5

I had the same problem. I solved it by

1. opening a dos command prompt

2. starting "c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" which modifies the PATH variable and sets some other variables

3. After that I started cygwin directly from this command prompt
   
   C:\cygwin64\bin\mintty.exe -i /Cygwin-Terminal.ico -
   
   

Now the PATH is properly formatted and msbuild is found ...

Hope this helps!

BTW: If it is complaining about the wrong version of msbuild maybe you have to install Visual Studio Express 2010. Or as I read elsewhere you can use VisualStudioExpress2012 and set VS100COMNTOOLS pointing to VS110COMNTOOLS.

Answer 6

I found the solution to this problem. I have VS 2012 on Windows 8.1 and faced this exact issue. I checked out the pom.xml under hadoop-common for that execution id and ran the following command

F:\HDP>msbuild F:\HDP\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln /nologo/p:Configuration=Release;OutDir=bin

It gave me the following exception

C:\Program Files(x86)\MSBuild\Microsoft.Cpp\v4.0\V110\Microsoft.Cpp.Platform.targets(42,5): error MSB8020: The builds tools for Visual Studio 2010 (Platform Toolset ='v100') cannot be found.
To build using the v100 build tools, either click the Project menu or right-click the solution, and then select "Update VC++ Projects...". Install Visual Studio 2010 to build using the Visual Studio 2010 build tools.
[F:\HDP\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj]

Since I do not have Visual Studio 2010 on my machine, the problem had to be in the solution. I opened the solution in VS 2012 and compiled it once. Executing maven again gave me another error but this time pointing to hadoop-common\src\main\native\native.sln. I opened that on VS 2012 too and no error.

Hope this helps

Thanks Rohit

Answer 7

It seems that there is problem with path..

add the msbuild path to Environment Variable Path