CODEWITHSUNDEEP
javacachinghadoopdistributed
Antoni
Antoni

Reputation: 2632

Can Hadoop Distributed cache contain folders or only files?

For one project I need to make available to the mappers several configuration files, but they are contained in different folders. Folder structure must be preserved.

It is possible to create a directory tree inside the distributed cache, and make accessible in the distributed cache the files in particular positions of that folder tree?

Thanks,

Upvotes: 1

Views: 1391

Answers (1)

cabad
cabad

Reputation: 4575

Take a look at the API documentation. It is very clear what you can or cannot do. You just add files to the cache, no path included. You can, however, include an archive and that archive can have a directory structure which will be recreated in the client.

Edit: Added the following info regarding user case.

Will all your mappers read all the configuration files, or just a subset of them? If it's the latter, then you should consider just creating the folder structure in HDFS. If different mappers are reading different files, then there wouldn't be an advantage in using the distributed cache.

Upvotes: 1

Related Questions