CODEWITHSUNDEEP
javamysqlhadoopguavahive
Pradeep
Pradeep

Reputation: 243

Issue while running hive job for giraph for graph analysis

we are try to run hive jobs on giraph, to do graph analysis, but every time we just keep bumping into new errors that we have never seen before.. and even after trying for hours we find no solutions.. now the we getting a new error..

we used following command to run hive job

sudo -u hdfs hive --service jar \
    giraph-hcatalog-0.2-SNAPSHOT-jar-with-dependencies.jar \
    org.apache.giraph.io.hcatalog.HiveGiraphRunner \
    -vertexClass org.apache.giraph.vertex.MutableVertex \
    -vertexInputFormatClass org.apache.giraph.io.hcatalog.HCatalogVertexInputFormat \
    -vertexOutputFormatClass org.apache.giraph.io.hcatalog.HCatalogVertexOutputFormat \
    -w 1 -vi giraph_input -o giraph_output \
    HIVE_OPTS="-hiveconf javax.jdo.option.ConnectionURL=jdbc:mysql://localhost/metastore 
               -hiveconf javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
               -hiveconf javax.jdo.option.ConnectionUserName=root
               -hiveconf javax.jdo.option.ConnectionPassword=root
               -hiveconf datanucleus.autoCreateSchema=false
               -hiveconf datanucleus.fixedDatastore=true"

is there anything wrong in i/p..? just wanted to run hive job using hivegiraphrunner, with hcat inputformatvertex and outformatvertex

hadoop, hive, mysql, are working, and runnig properly (cofigured hadoop_env , hive-site) using hadoop-0.20.2 hadoop-hive (cloudera)

giraph jars were a successful build

is their something that i missed, or should configure

any suggestion will be a great help..!!

getting following exceptions.. 

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutorService;
    at com.google.common.cache.LocalCache.<clinit>(LocalCache.java:156)
    at com.google.common.cache.LocalCache$LocalManualCache.<init>(LocalCache.java:4765)
    at com.google.common.cache.CacheBuilder.build(CacheBuilder.java:821)
    at org.apache.hcatalog.common.HiveClientCache.<init>(HiveClientCache.java:89)
    at org.apache.hcatalog.common.HCatUtil.getHiveClient(HCatUtil.java:537)
    at org.apache.hcatalog.mapreduce.HCatUtils.getInputJobInfo(HCatUtils.java:75)
    at org.apache.giraph.io.hcatalog.GiraphHCatInputFormat.setVertexInput(GiraphHCatInputFormat.java:81)
    at org.apache.giraph.io.hcatalog.HiveGiraphRunner.run(HiveGiraphRunner.java:174)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.giraph.io.hcatalog.HiveGiraphRunner.main(HiveGiraphRunner.java:147)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
    enter code here

Upvotes: 0

Views: 811

Answers (2)

Jugal
Jugal

Reputation: 1

You can import the guava from /hive/lib.And if any other dependency arise add them to

Upvotes: 0

Frank Pavageau
Frank Pavageau

Reputation: 11725

Giraph has a dependency on guava 12.0 (cf. its POM, lines 752-756), which explains that MoreExecutors.sameThreadExecutor() (introduced in 10.0) is not found if you already have r06 in hive/lib and Hive directly loads giraph in its JVM.

You could try upgrading the jar in Hive, but with such distance between the 2 versions, it could well be incompatible. In that case, I guess you may have to upgrade Hive, though I'm not a Hadoop user and don't know the details.

Upvotes: 1

Related Questions