How to copy files from a remote server to hdfs location

Question

I want to copy files from a remote server using sftp to an hdfs location directly without copying the files to local. The hdfs location is a secured cluster. Please suggest if this is feasible and how to proceed in that case. Also I would want to know if there is any other way to connect and copy apart from sftp.

Answer 1

I think the most convenient way (given that your remote machine is able to connect to the hadoop cluster) is to make that remote machine act as an HDFS client. Just ssh to that machine, install the hadoop distribution, configure it properly, then run:

hadoop fs -put /local/path /hdfs/path