Installing Hadoop's Eclipse Plugin

Question

I have such a nightmare with the Hadoop's Eclipse plugin. First of all the latest Hadoop version (which is 1.1.1) doesn't even include the Eclipse plugin jar file. version 1.0.4 has the plugin, but it doesn't work. Earlier versions of Hadoop such as 0.22.0 have the plugin and it works, but then these versions of Hadoop are old, and I want to work wit the latest version. So, what is the deal with Eclipse plugin for Hadoop? Why latest versions either don't have the plugin, or if they do, then the plugin doesn't work? Does everyone has so much problem with this plugin? Should I give up on this plugin?

Thanks, Shannon

Answer 1

Well, combined with hs3180's answer, here is what I actually did in configuring the compiling.

1. In ${eclipse-plugin-src}/build.xml, use the following directives to include required jars by the eclipse-plugin. Note here that, instead of from ${hadoop.root}/build, copying of the hadoop-core-${version}.jar is now from ${hadoop.root} directly, because if you are using the compiled version of hadoop, the ${hadoop.root}/build folder would be actually empty. Copying of the commons-cli-${commons-cli.version}.jar is now from ${hadoop.root}/lib for the same reason.

   <!-- <copy file="${hadoop.root}/build/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/> -->
   <!-- <copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/> -->
   <copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
   <copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/commons-configuration-${commons-configuration.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/commons-httpclient-${commons-httpclient.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/commons-lang-${commons-lang.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/jackson-core-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   <copy file="${hadoop.root}/lib/jackson-mapper-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
   

2. Modify the ${eclipse-plugin-src}/META-INF/MANIFEST.MF file to accommodate the changes made to ${eclipse-plugin-src}/build.xml.

   Bundle-ClassPath: classes/,
    lib/hadoop-core.jar,
    lib/commons-cli-1.2.jar,
    lib/commons-configuration-1.6.jar,
    lib/jackson-core-asl-1.8.8.jar,
    lib/commons-httpclient-3.0.1.jar,
    lib/jackson-mapper-asl-1.8.8.jar,
    lib/commons-lang-2.4.jar
   

3. Modify the ${hadoop.root}/src/contrib/build-contrib.xml file at two places. The first is to set up the properties needed for the plugin compiling. And the second is to make sure jars, especially hadoop-core-1.0.4.jar, in the ${hadoop.root} folder are visible to javac, because the eclipse-plugin references the hadoop classes. This setup is different from the first two in purpose and thus not dispensable.

   <!-- Properties added for compiling eclipse-plugin -->
   <!-- http://yiyujia.blogspot.com/2012/11/build-hadoop-eclipse-plugin-from-source.html -->
   <property name="eclipse.home" location="/Users/xuj/Downloads/eclipse/"/>
   <property name="version" value="1.0.4"/>
   <property name="commons-cli.version" value="1.2"/>
   
   <!-- the normal classpath -->
   <path id="contrib-classpath">
     <fileset dir="${hadoop.root}">
       <include name="*.jar" />
     </fileset>
     <!-- more path elements go here -->
   </path>
   

When all files correctly configured, call 'ant jar' from ${eclipse-plugin-src}/ in the console shall be sufficient for the rest.

Answer 2

I installed eclipse plugin for hadoop-1.0.4 yesterday with difficulty, and tested it successful.

The reason for plugin not working is that the jar package lost some libs :

• commons-cli-1.2.jar
• commons-configuration-1.6.jar
• jackson-core-asl-1.8.8.jar
• jackson-mapper-asl-1.8.8.jar
• commons-httpclient-3.0.1.jar
• commons-lang-2.4.jar

you could cp this jars from ${hadoop}/lib to ${jar}/lib, and don't forget modifying MANIFEST.

For convenience, I add some code to ${eclipse-plugin-src}/build.xml target jar

<copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-configuration-${commons-configuration.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-httpclient-${commons-httpclient.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-lang-${commons-lang.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-core-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-mapper-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>

and modified MANIFEST.MF in ${eclipse-plugin-src}/META-INF

undle-ClassPath: classes/,
 lib/hadoop-core.jar,
 lib/commons-cli-1.2.jar,
 lib/commons-configuration-1.6.jar,
 lib/jackson-core-asl-1.8.8.jar,
 lib/commons-httpclient-3.0.1.jar,
 lib/jackson-mapper-asl-1.8.8.jar,
 lib/commons-lang-2.4.jar

rebuild eclipse-plugin, and HAVE FUN!