CODEWITHSUNDEEP
javahadoopmapreduce
Boolean
Boolean

Reputation: 14664

Adding multiple files to Hadoop distributed cache?

I am trying to add multiple files to hadoop distributed cache. Actually I don't know the file names. They will be named like part-0000*. Can someone tell me how to do that?

Thanks Bala

Upvotes: 5

Views: 3355

Answers (3)

Breakinen
Breakinen

Reputation: 619

I solved this problem although it maybe a bit late:

FileSystem fs = directoryPath.getFileSystem(getConf());
FileStatus[] fileStatus = fs.listStatus(directoryPath);
for (FileStatus status : fileStatus) {
    DistributedCache.addFileToClassPath(status.getPath(), conf);
}

Is this what you wanted to do?

Upvotes: 1

Dmytro Molkov
Dmytro Molkov

Reputation: 11

Nothing prevents you from programmatically getting the list of files if they all are in one directory and the adding them one by one, right? Or is your case different?

Upvotes: 0

Matthew Hegarty
Matthew Hegarty

Reputation: 4231

You can use either the hadoop -put or -copyFromLocal command:

hadoop fs -copyFromLocal /home/hadoop/outgoing/* /your/hadoop/dir

Upvotes: 2

Related Questions