Making Pig 0.12 work with hadoop 2.2.0

Question

I have hadoop 2.2.0 running on remote cluster and Pig 0.12 on a separate machine. I need to make Pig communicate with hadoop and the first steps seems to be to build Pig 0.12 with hadoop 2.2.0. Here is what I did:

1. In ivy/libraries.properties changed the hadoop-core.version, hadoop-common.version, hadoop-hdfs.version, hadoop-mapreduce.version to 2.2.0.
2. In ivy.xml replaced the hadoop-core dependency to hadoop-client dependency.
3. Build pig using "ant clean jar-all -Dhadoopversion=23"

When I run "pig" from command line I get the following error:

ERROR 2998: Unhandled internal error. org/apache/hadoop/hdfs/DistributedFileSystem

java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/DistributedFileSystem
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:173)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:122)
        at org.apache.pig.impl.PigContext.connect(PigContext.java:301)
        at org.apache.pig.PigServer.<init>(PigServer.java:222)
        at org.apache.pig.PigServer.<init>(PigServer.java:207)
        at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
        at org.apache.pig.Main.run(Main.java:538)
        at org.apache.pig.Main.main(Main.java:156)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.DistributedFileSystem
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Answer 1

Just building with command ant clean jar-all -Dhadoopversion=23 or ant jar-withouthadoop –Dhadoopversion=23 is not enough if you are using maven dependencies in your project. You will need to install the jar created by this in your local maven repo or use this dependency (notice "classifier" tag for hadoop2) in your pom.xml

<dependency>
  <groupId>org.apache.pig</groupId>
  <artifactId>pig</artifactId>
  <version>0.13.0</version>
  <classifier>h2</classifier>
</dependency>