CODEWITHSUNDEEP
javahadoophdfscloudera
user3013201
user3013201

Reputation: 21

Error while copying a file from HDFS to Windows machine

There is a Linux VM with Hadoop installed and running. And there is Java app running in Eclipse that retrieve data from HDFS. If I am copying file(s) to or from HDFS inside the VM everything works fine. But when i am running the app from my Windows physical machine I am getting the next exception:

WARN hdfs.DFSClient: Failed to connect to /127.0.0.1:50010 for block, add to 
deadNodes and continue. java.net.ConnectException: Connection refused: no further 
information. Could not obtain BP-*** from any node: java.io.IOException: 
No live nodes contain current block. Will get new block locations from namenode and retry

I can only retrieve list of files from HDFS. Seems that when retrieve data from data node it is connecting to my Windows localhost. Because when I made a tunnel in putty from my localhost to VM everything was fine.

Here is my Java code:

Configuration config = new Configuration();
config.set("fs.defaultFS", "hdfs://ip:port/");
config.set("mapred.job.tracker", "hdfs://ip:port");
FileSystem dfs = FileSystem.get(new URI("hdfs://ip:port/"), config, "user");
dfs.copyToLocalFile(false, new Path("/tmp/sample.txt"),newPath("D://sample.txt"), true);

How can it be fixed? Thanks.

P.S. This error occurs when I am using QuickStart VM from Cloudera.

Upvotes: 2

Views: 2978

Answers (3)

Abhay Dandekar
Abhay Dandekar

Reputation: 1300

Whenever you start a VM, it gets its own I.P. Something like 192.x.x.x or 172.x.x.x.

Using 127.0.0.1 for HDFS wont help when you are executing from your windows box, because this is mapped to local i.p. So, if you are using 127.0.0.1 from your windows machine, it will think that your HDFS is running on windows machine. This is why your connection is failing.

Find the i.p that is associated with your VM. Here is a link to get that if you are using Hyper-V. http://windowsitpro.com/hyper-v/quickly-view-all-ip-addresses-hyper-v-vms

Once you get the VMs I.P, use it in the application.

Upvotes: 1

Chris White
Chris White

Reputation: 30089

Your DataNode is advertising its address to the NameNode as 127.0.0.1. You need to re-configure your Pseudo distributed cluster such that the nodes use externally available addresses (hostnames or IP addresses) when opening socket services.

I imagine if you run a netstat -atn on your VM, you'll see the Hadoop ports bound to 127.0.0.1 rather than 0.0.0.0 - this means they will only accept internal connections.

You need to look at your VM's /etc/hosts configuration file and ensure hostname doesn't have an entry resolving to 127.0.0.1.

Upvotes: 1

Animesh Raj Jha
Animesh Raj Jha

Reputation: 2724

You need to change the ip. First go to linux VM and in its terminal find the IP address of your VM.

Command to see the ip address in linux VM is below

ifconfig

Then in your code change the ip address to the IP thats shown in your linux VM.

Upvotes: 0

Related Questions