CODEWITHSUNDEEP
hadoophdfs
scoder
scoder

Reputation: 2611

hdfs or hadoop command to sync the files or folder between local to hdfs

I have a local files which gets added daily so I want to sync these newly added files to hdfs.

I tried below command but all are complete copy, I want some command which copies only newly added files 

$ hdfs dfs -cp /home/user/files/* /data/files/*

Upvotes: 0

Views: 3458

Answers (1)

StrangerThinks
StrangerThinks

Reputation: 248

You can use hsync. https://github.com/alexholmes/hsync

Its Alex's custom package and perhaps useful on a dev box but could be hard to deploy on production environment. I am looking for a similar solution but for now this seems to be closest. Other option is to write your own shell script to compare source/target file times and then overwrite newer files only.

Upvotes: 1

Related Questions