CODEWITHSUNDEEP
pythonhadoop
lmwilco1
lmwilco1

Reputation: 1

Installing pydoop 1.0.0-rc1 on CDH 5.3.0

I am trying to upgrade to pydoop 1.0.0-rc1 on a linux server running CDH 5.3.0. I was able to complete the installation on another environment running CDH 5.1.2. I run the command "pip install --proxy .... -upgrade pydoop==1.0.0-rc1" and the package is downloaded and the c++ compiling completes. When it gets to the java build it fails:

running build_scripts                                                                                                  
creating build/scripts-2.7                                                                                             
copying and adjusting scripts/pydoop -> build/scripts-2.7                                                              
changing mode of build/scripts-2.7/pydoop from 644 to 755                                                              
hadoop_home: '/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars'                                                   
hadoop_version: '2.5.0-cdh5.3.0'                                                                                       
java_home: '/usr/java/latest'                                                                                          
Building java code for hadoop-2.5.0-cdh5.3.0                                                                           
Compiling Java classes                                                                                                 
src/v1/it/crs4/pydoop/NoSeparatorTextOutputFormat.java:31: cannot find symbol                                          
symbol  : class JobConf                                                                                                
location: package org.apache.hadoop.mapred                                                                             
import org.apache.hadoop.mapred.JobConf;                                                                               
                               ^                                                                                       
src/v1/it/crs4/pydoop/NoSeparatorTextOutputFormat.java:32: cannot find symbol                                          
symbol  : class RecordWriter                                                                                           
location: package org.apache.hadoop.mapred                                                                             
import org.apache.hadoop.mapred.RecordWriter;                                                                          
                               ^                                                                                       
src/v1/it/crs4/pydoop/NoSeparatorTextOutputFormat.java:33: cannot find symbol

Based on the logging it has the location of the hadoop libraries so I am at loss as what to try next?

Does this version of pydoop support CDH 5.3.0? Any suggestions how to fix this?

Thanks for any suggestions.

Louis

Upvotes: 0

Views: 171

Answers (1)

mdrio
mdrio

Reputation: 11

I am a Pydoop developer. I tested cdh 5.3.0 on the last pydoop release (1.0.0) and it worked. Maybe the best thing is to remove the previous installation and re run: sudo pip install pydoop

Upvotes: 1

Related Questions