CODEWITHSUNDEEP
hadoopreplication
dverma
dverma

Reputation: 11

How to make hadoop replication factor change for a file persistent

I have a query regarding hadoop replication.

I changed the replication factor of files in a particular directory in hadoop using the below mentioned command:

hadoop dfs -setrep -R 3 -w /data/routing

It was successful and it set the replication factor of all the files in this directory to 3. However any new file being written under this directory continued to have default replication factor, i.e. 2.

Is there any option to make this change persistent?

Actually I want all the new files being written under this directory only to always have the replication factor as 3, irrespective of what is the default replication factor.

Upvotes: 1

Views: 2670

Answers (1)

Chris White
Chris White

Reputation: 30089

If you want something other than the default replication factor, you'll need to explicitly set the replication factor when you create the file(s).

Are the files under the directory created via a mapreduce job, or manually using some other process?

For mapreduce, just change/set the job configuration value for the default replication factor - dfs.replication. If you're creating the file(s) manually in some java code, then look at the API for FileSystem.create(Path, short)

Upvotes: 1

Related Questions