Couchbase/hadoop connector: sqoop job FAILED "Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected"

Question

My Configuration:

• CouchBase server 2.0
• Sqoop 1.4.2 (compile for hadoop version 2.0.0)
• Stack Hadoop CDH4.1.2

I want to use the CouchBase/Hadoop connector (http://www.couchbase.com/develop/connectors/hadoop), but when I launch an import I have the following error:

sqoop-import --connect http://localhost:8091/pools --table DUMP
13/01/07 11:00:34 INFO tool.CodeGenTool: Beginning code generation
13/01/07 11:00:34 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop
Note: /tmp/sqoop-agent/compile/fccb413cf9eec92f9a716320cf268a0a/DUMP.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/01/07 11:00:42 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-agent/compile/fccb413cf9eec92f9a716320cf268a0a/DUMP.jar
13/01/07 11:00:42 INFO mapreduce.ImportJobBase: Beginning import of DUMP
13/01/07 11:00:45 WARN util.Jars: No such class couchbase doesn't use a jdbc driver available.
13/01/07 11:00:50 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2013-01-07 11:00:55.065 INFO net.spy.memcached.MemcachedConnection:  Added {QA sa=/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-01-07 11:00:55.098 INFO net.spy.memcached.MemcachedConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@41e22632
2013-01-07 11:00:56.266 INFO net.spy.memcached.MemcachedConnection:  Shut down memcached client
13/01/07 11:02:39 INFO mapred.JobClient: Running job: job_201212111702_0131
13/01/07 11:02:40 INFO mapred.JobClient:  map 0% reduce 0%
13/01/07 11:02:59 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000001_0,     Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:02 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000002_0, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:02 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000000_0, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:09 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000003_0, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:12 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000001_1, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:13 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000002_1, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:20 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000000_1, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:24 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000003_1, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:24 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000001_2, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:30 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000002_2, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:33 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000000_2, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:39 INFO mapred.JobClient: Job complete: job_201212111702_0131
13/01/07 11:03:40 INFO mapred.JobClient: Counters: 7
13/01/07 11:03:40 INFO mapred.JobClient:   Job Counters 
13/01/07 11:03:40 INFO mapred.JobClient:     Failed map tasks=1
13/01/07 11:03:40 INFO mapred.JobClient:     Launched map tasks=14
13/01/07 11:03:40 INFO mapred.JobClient:     Rack-local map tasks=14
13/01/07 11:03:40 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=74621
13/01/07 11:03:40 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=0
13/01/07 11:03:40 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/01/07 11:03:40 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/01/07 11:03:40 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 174.4617 seconds (0 bytes/sec)
13/01/07 11:03:40 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
13/01/07 11:03:40 INFO mapreduce.ImportJobBase: Retrieved 0 records.
13/01/07 11:03:40 ERROR tool.ImportTool: Error during import: Import job failed!

Does anyone have used this connector with the same configuration and have succeeded to make it works?

Thx for your help!

Madeleine

Answer 1

This was fixed in version CDH4.2.

http://www.cloudera.com/content/cloudera/en/documentation/cdh4/v4-2-0/CDH4-Release-Notes/cdh4ki_topic_2_9.html

Answer 2

Hadoop has gone through a huge code refactoring from Hadoop 1.0 to Hadoop 2.0 (correspondingly from CDH3 to CDH4). One side effect is that code compiled against Hadoop 1.0 (CDH3) is not compatible with Hadoop 2.0 (CDH4) and vice-versa. However source code is compatible and thus one just need to recompile code with target Hadoop distribution.

The exception "Found interface X, but class was expected" is very common when you're running code that is compiled for Hadoop 1.0 (CDH3) on Hadoop 2.0 (CDH4) or vice-versa.

Exactly the same constraint applies to connectors. There is no explicit information for which Hadoop version CouchBase connector was compiled, so I would guess that it was against Hadoop 1.0 and thus it's not working on Hadoop 2.0 (CDH4).