Sync File Modification Time Across Multiple Directories

Question

I have a computer A with two directory trees. The first directory contains the original mod dates that span back several years. The second directory is a copy of the first with a few additional files. There is a second computer be which contains a directory tree which is the same as the second directory on computer A (new mod times and additional files). How update the files in the two newer directories on both machines so that the mod times on the files are the same as the original? Note that these directory trees are in the order of 10s of gigabytes so the solution would have to include some method of sending only the date information to the second computer.

Answer 1

I used the following Python scripts instead.

Python scripts run much faster than an approach creating new processes for each file (like using find and stat). The solution below also works in case of timezone differences between systems, as it uses UTC times. It also works with paths containing spaces (but not paths containing newline!). It doesn't set times for symlinks, because the operating system provides no mechanism to modify the timestamp of a symlink, but in a file manager the time of the file the symlink points at is shown instead anyway. It uses a maxTime parameter to avoid resetting dates for files that are actually modified after copying from the original directory.

listMTimes.py:

import os
from datetime import datetime
from pytz import utc

for dirpath, dirnames, filenames in os.walk('./'):
    for name in filenames+dirnames:
        path = os.path.join(dirpath, name)
        # Avoid symlinks because os.path.getmtime and os.utime get and
        # set the time of the pointed file, and in the new directory,
        # the link may have been redirected.
        if not os.path.islink(path):
            mtime = datetime.fromtimestamp(os.path.getmtime(path), utc)
            print(mtime.isoformat()+" "+path)

setMTimes.py:

import datetime, fileinput, os, sys, time
import dateutil.parser
from pytz import utc

# Based on
# http://stackoverflow.com/questions/6999726/python-getting-millis-since-epoch-from-datetime
def unix_time(dt):
    epoch = datetime.datetime.fromtimestamp(0, utc)
    delta = dt - epoch
    return delta.total_seconds()

if len(sys.argv) != 2:
    print('Syntax: '+sys.argv[0]+' <maxTime>')
    print('  where <maxTime> an ISO time, e. g. "2013-12-02T23:00+02:00".')
    exit(1)

# A file with modification time newer than maxTime is not reset to
# its original modification time.
maxTime = unix_time(dateutil.parser.parse(sys.argv[1]))

for line in fileinput.input([]):
    (datetimeString, path) = line.rstrip('\r\n').split(' ', 1)
    mtime = dateutil.parser.parse(datetimeString)
    if os.path.exists(path) and not os.path.islink(path):
        if os.path.getmtime(path) <= maxTime:
            os.utime(path, (time.time(), unix_time(mtime)))

Usage: in the first directory (the original) run

python listMTimes.py >/tmp/original_dates.txt

Then in the second directory (a copy of the original, possibly with some files modified/added/deleted) run something like this:

python setMTimes.py 2013-12-02T23:00+02:00 </tmp/original_dates.txt

Answer 2

The answer by Paul is partly correct, rsync is able to do this, however with different parameters. The correct command is

rsync -Prt --size-only original_dir copy_dir

where -P enables partial transfers and displays a progress indicator, -r recurses through subdirectories, -t preserves time stamps and --size-only doesn't transfer files that match in size.

Answer 3

I think rsync (with the right options) will do this - it claims to only send file differences, so presumably will work out that there are no differences to be transferred.

--times preserves the modification times, which is what you want.

See (for instance) http://linux.die.net/man/1/rsync

Also add -I, --ignore-times don't skip files that match size and time

so that all files are "transferred' and trust to rsync's file differences optimisation to make it "fairly efficient" - see excerpt from the man page below.

-t, --times This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing -t or -a will cause the next transfer to behave as if it used -I, causing all files to be updated (though the rsync algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using -t).

Answer 4

The following command will make sure that TEST2 gets the same date assigned that TEST1 has

touch -t `stat -t '%Y%m%d%H%M.%S' -f '%Sa' TEST1` TEST2

Now instead of using hard-coded values here, you could find the files using "find" utility and then run touch via SSH on the remote machine. However, that means you may have to enter the password for each file, unless you switch SSH to cert authentication. I'd rather not do it all in a super fancy one-liner. Instead let's work with temp files. First go to the directory in question and run a find (you can filter by file type, size, extension, whatever pleases you, see "man find" for details. I'm just filtering by type file here to exclude any directories):

find . -type f -print -exec stat -t '%Y%m%d%H%M.%S' -f '%Sm' "{}" \; > /tmp/original_dates.txt

Now we have a file that looks like this (in my example there are only two entries there):

# cat /tmp/original_dates.txt 
./test1
200809241840.55
./test2
200809241849.56

Now just copy the file over to the other machine and place it in the directory (so the relative file paths match) and apply the dates:

cat original_dates.txt | (while read FILE && read DATE; do touch -t $DATE "$FILE"; done)

Will also work with file names containing spaces.

One note: I used the last "modification" date at stat, as that's what you wrote in the question. However, it rather sounds as if you want to use the "creation" date (every file has a creation date, last modification date and last access date), you need to alter the stat call a bit.

'%Sm' - last modification date
'%Sc' - creation date
'%Sa' - last access date

However, touch can only change the modification time and access time, I think it can't change the creation time of a file ... so if that was your real intention, my solution might be sub-optimal... but in that case your question was as well ;-)

Answer 5

The other approach you could try is to attach the remote directory using NFS and then copy the times using find and touch -r.

Answer 6

I would go through all the files in the source directory tree and gather the modification times from them into a script that I could run on the other directory trees. You will need to be careful about a few 'gotchas'. First, make sure that your output script has relative paths, and make sure you run it from the proper target directory, which should be the root directory of the target tree. Also, when changing machines make sure you are using the same timezone as you were on the machine where you generated the script.

Here's a Perl script I put together that will output the touch commands needed to update the times on the other directory trees. Depending on the target machines, you may need to tweak the date formats or command options, but this should give you a place to start.

#!/usr/bin/perl

my $STARTDIR="$HOME/test";

chdir $STARTDIR;
my @files = `find . -type f`;
chomp @files;

foreach my $file (@files) {
   my $mtime = localtime((stat($file))[9]);
   print qq(touch -m -d "$mtime" "$file"\n);
}