CODEWITHSUNDEEP
javaspringhadoopmapreducespring-data
user3222372
user3222372

Reputation: 382

Hadoop getting Connection Refused exception while starting job from Spring batch admin

I am trying to trigger hadoop mapreduce job from spring Admin but getting below error. Spring Admin is running on wasce.

Job Config:

<hdp:configuration>
    fs.defaultFS=hdfs://localhost:8020
    mapred.job.tracker=localhost:8021
</hdp:configuration>

<hdp:job id="mr-my-job" 
        input-path="/data/input/"
        output-path="/data/output/"
        jar-by-class="org.test.Main" 
        mapper="org.test.Test1$Map"
        combiner="org.test.Test1$Combiner"
        reducer="org.test.Test1$ReduceFromCombiner" />

Please advice. When I run this job by packaging a jar and run using hadoop command it works fine but give an error while running through Admin. sudo -u hdfs hadoop jar test.jar org.Main

Caused by: java.net.ConnectException: Call From <server>/<server_ip> to localhost:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
        at org.apache.hadoop.ipc.Client.call(Client.java:1413)
        at org.apache.hadoop.ipc.Client.call(Client.java:1362)
        at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231)
        at org.apache.hadoop.mapred.$Proxy203.getStagingAreaDir(Unknown Source)
        at org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1340)
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:954)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:948)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:582)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:612)
        at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:199)
        ... 24 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:604)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:699)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1461)
        at org.apache.hadoop.ipc.Client.call(Client.java:1380)

Upvotes: 0

Views: 812

Answers (1)

stevel
stevel

Reputation: 13480

We have a special page on hadoop related to this. Please follow the link in the stack trace you saw: http://wiki.apache.org/hadoop/ConnectionRefused

Can I just say that I put a lot of effort into trying to improve connectivity diagnostics, including the reporting of destination hosts and ports, the listing of URLs with debugging advice, and indeed, the writing and maintenance of those pages. It always makes me very sad when people don't follow the links, as I feel my effort has been wasted. Please, when you see such URLs in stack traces —do at least take a cursory look at the URL referenced.

Upvotes: 2

Related Questions