CODEWITHSUNDEEP
javahadoopmapreduce
IMAN Sharafoddin
IMAN Sharafoddin

Reputation: 23

How we can force many mappers read one specific file (same data) in hadoop?

i want to write a program that many mappers read one 1 file that is a graph and all do processing on that graph, that file is about 14 kb if i run in normal way just one mapper will be mapped for that file how i can do that? Best regards.

Upvotes: 2

Views: 206

Answers (1)

Praveen Sripati
Praveen Sripati

Reputation: 33495

Put the file in HDFS and read the file using HDFS API in the mapper function and do the processing on it. You can also try DistributedCache instead of putting the file in HDFS.

This scenario is also applicable when N simulations have to be done on a data set. The simulation factors will be the input to the map function and the actual data is read in the map function.

Upvotes: 1

Related Questions