Access HDFS HADOOP using APACHE Web Server, Linux CentOS

Question

If I have an apache web server as a directory how I can access to HDFS Cluster for upload and modify files, what is the configuration i want to do?

many thanks

Answer 1

I dont understand what you are acyually asking , i assume you are designing some web application, and you want HDFS access from it , right ? To manipulate HDFS , we use Hadoop API. So , you can easily access HDFS using Hadoop API. Use java Apache wickets for developing your web application, because it will let you use java with HTML for designing a web site. you can then easily import Hadoop API classes to access to HDFS.

Another way is , ifyou dont want to use Hadoop API to access HDFS , we generally use terminal to manipulate HDFS. for example :

hadoop fs -put <file name> <dir name> 

to upload data in HDFS. use 'ProcessBuilder' class of java , through which you can invoke any shell command from inside the java program. I give ypu code to invoke shell commands from inside the java program :

protected final String executeCommand2(String [] parts)
{
int len = parts.length;
ProcessBuilder builder;
if(len==0) return null;
else if(len==1) builder = new ProcessBuilder(parts[0]);
else if(len==2) builder = new ProcessBuilder(parts[0],parts[1]);
else if(len==3) builder = new ProcessBuilder(parts[0],parts[1],parts[2]);
else if(len==4) builder = new ProcessBuilder(parts[0],parts[1],parts[2],parts[3]);
else if(len==5) builder = new ProcessBuilder(parts[0],parts[1],parts[2],parts[3],parts[4]);
else if(len==6) builder = new ProcessBuilder(parts[0],parts[1],parts[2],parts[3],parts[4],parts[5]);
else if(len==7) builder = new ProcessBuilder(parts[0],parts[1],parts[2],parts[3],parts[4],parts[5],parts[6]);
else if(len==8) builder = new ProcessBuilder(parts[0],parts[1],parts[2],parts[3],parts[4],parts[5],parts[6],parts[7]);
else return null;
        builder.redirectErrorStream(true);
        Process process = null;
        String newstr="" , oldstr = "";
        try{
             process = builder.start();
             InputStream is = process.getInputStream();
             InputStreamReader isr = new InputStreamReader(is);
             BufferedReader br = new BufferedReader(isr);
             while ((newstr = br.readLine()) != null)
             {
                 oldstr = oldstr+"\n"+newstr;

             }
                 process.waitFor();
            }catch(Exception e)
            {
                newstr = "Exception = " + e;
                oldstr = oldstr+"\n"+newstr;

            }
        int exitCode = process.exitValue();
        if(exitCode == 0)
            {
                 newstr="Successfull termination "+exitCode ;
                 oldstr = oldstr+"\n"+newstr;
            }
        else
            {
                 newstr="abrupt termination "+ exitCode;
                 oldstr = oldstr+"\n"+newstr;
             }
        newstr="##################################";
         oldstr = oldstr+"\n"+newstr;
        if(oldstr.length() > 2000)
           oldstr = "";
        return oldstr;
    } // function ends here

invoke the above function as follows :

String resultLines = executeCommand2(new String[]{
                    "sh",
        "<absolute path to hadoop script present in $HADOOP_HOME/conf",
                    "fs",
                    "-put",
                    "absolute path to your yourfile.txt",
                    "input HDFS dir"
                    });

resultLines will contain the same text as would have had appeared on the shell as a result of invoking the command there. you can invoke any shell command having upto 7 arguments. ofcourse you can modify the function to accept shell command with more no of arguments. hope it hep you.

Note that, if you are using tomcat apache web server , then login the server with the same username same as the username of your linux account so that you have permissions to access scripts on your local drive. To do so , edit tomcat-users.xml file in /var/lib/tomcat6/conf firectory. add the following lines:

 <role rolename="manager"/>
 <role rolename="admin"/>
<user username="your linux user name" password="your passwd" roles="admin,manager"/>

thats it.

Also note that, if you use processbuilder class mechanism to build your application, then your application will become linux specific and will not run on windows machines.