CODEWITHSUNDEEP
servletshadoop
Shailesh
Shailesh

Reputation: 403

Submitting Hadoop Job on remote Hadoop cluster from web application (servlet)

I have configured a Hadoop cluster with 2 nodes of Linux boxes and I have a servlet application running on Tomcat on third Linux box. My goal is to submit Hadoop from web app. When I submit job from Eclipse then I am able to submit job on Hadoop remotely. I am getting following exception in case of servlet. Please suggest

'java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1414) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231) at $Proxy11.getSystemDir(Unknown Source) at org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:141) at org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:139) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:139) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1040) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) at servlets.JobDriver.doProcess(JobDriver.java:112) at servlets.JobDriver.doGet(JobDriver.java:42) at javax.servlet.http.HttpServlet.service(HttpServlet.java:620) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:662)'

Upvotes: 0

Views: 604

Answers (1)

ranganath gurram
ranganath gurram

Reputation: 1

From the logs it seems the user from your servlet making the job submission has no permission to submit the job to the cluster:

$Proxy11.getSystemDir(Unknown Source) at 
org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:141) at 
org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:139) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at 
org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:139) at  
org.apache.hadoop.mapreduce.Job.submit(Job.java:1040) at

Make sure only the admin group submits from servlet by checking the servlet Session.

Upvotes: 0

Related Questions