CODEWITHSUNDEEP
javahadoophdfs
Liondancer
Liondancer

Reputation: 16469

Running hdfs -text command within mapper and using output

Is there a way to run hadoop commands in the mapper?

For example I want to run 

hdfs dfs -text /path/to/file

and I will use that output for my mapper logic. Or is there an alternative?

Such as running the command hdfs dfs -text /path/to/file as a subprocess with Java

Upvotes: 0

Views: 161

Answers (1)

Jeremy Beard
Jeremy Beard

Reputation: 2725

Better is to use the Java HDFS API to open an HDFS file and read the contents into a variable that you can use.

For example:

InputStream is = FileSystem.get(yourConfigurationObject).open(new Path("/path/to/file"));

Upvotes: 1

Related Questions