CODEWITHSUNDEEP
node.jshadoophdfswebhdfs
shihack
shihack

Reputation: 55

Not able to access files in hadoop cluster

I was trying to read the file present in hadoop cluster through the following code. The default port used is 9000. (since at 50700, it is not getting connected)

//webhdfs-read-tests.js
// Include webhdfs module
var WebHDFS = require('webhdfs');
var fs = require('fs');
// Create a new
var hdfs = WebHDFS.createClient({
    user: 'ubuntu', // Hadoop user
    host: 'localhost', // Namenode host
    port: 9000 // Namenode port
});

// Initialize readable stream from HDFS source
var remoteFileStream = hdfs.createReadStream('/sample.txt');

// Variable for storing data
var data = new Buffer.alloc(10);

remoteFileStream.on('error', function onError(err) {
    // Do something with the error
    console.log(err);
    //console.log('')
});

remoteFileStream.on('data', function onChunk(chunk) {
    // Concat received data chunk
    data = Buffer.concat([data, chunk]);
    console.log('data', data);
});

remoteFileStream.on('finish', function onFinish() {
    // Upload is done
    // Print received data
    console.log(data.toString());
});

The cluster looks like:

hadoop fs -ls -R /
2022-12-21 10:52:30,393 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
drwxr-xr-x   - ubuntu supergroup          0 2022-12-21 08:28 /files
-rw-r--r--   1 ubuntu supergroup        169 2022-12-21 08:28 /files/sample.txt
-rw-r--r--   1 ubuntu supergroup        169 2022-12-21 10:24 /sample.txt
drwxr-xr-x   - ubuntu supergroup          0 2022-12-21 08:30 /uploadedFiles

Also, the jps command gives:

jps
47232 ResourceManager
52208 Jps
46768 DataNode
46598 NameNode
47031 SecondaryNameNode
47405 NodeManager

After running the code, I got the following error message:

Error: Invalid request path or webhdfs is not enabled: /webhdfs/v1/sample.txt
    at WebHDFS._parseError (/home/ubuntu/node_modules/webhdfs/lib/webhdfs.js:89:16)
    at Request.onData (/home/ubuntu/node_modules/webhdfs/lib/webhdfs.js:656:32)
    at Request.emit (events.js:198:13)
    at IncomingMessage.<anonymous> (/home/ubuntu/node_modules/request/request.js:1073:12)
    at IncomingMessage.emit (events.js:198:13)
    at IncomingMessage.Readable.read (_stream_readable.js:505:10)
    at flow (_stream_readable.js:974:34)
    at resume_ (_stream_readable.js:955:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)

Can someone help me with this?

Upvotes: 0

Views: 124

Answers (1)

shihack
shihack

Reputation: 55

The solution is to use the correct port number.

  • 9000 is the port of the namenode.
  • 9870 is the port of webhdfs-> correct port.

Upvotes: 0

Related Questions