CODEWITHSUNDEEP
linuxbashrsync
machinery
machinery

Reputation: 6290

Let rsync take a snapshot of files before copying

I have the following bash script. In the script I use rsync to copy files from a source to a destination. In the first call of rsync I copy all the files and in the second call I double-check the files and if the checksum is valid the copied files are deleted in the source.

#!/bin/bash
set -e
rsync --info=progress2 -r --include='database/session_*.db' --exclude 'database/session*' /local/data/ /import/myNas/data
rsync --info=progress2 -r --include='database/session_*.db' --exclude 'database/session*' --checksum --remove-source-files /local/data/ /import/myNas/data

The problem now is that while rsync is running new files are written to /local/data. I would like that rsync takes a snapshot of the list of files in source (/local/data) when it runs the first time and then only copies these files. In the second run rsync should then also only run on these files from the snapshot (i.e. calculate the checksum and then delete the files). That means the new added files should not be touched.

Is this possible?

Upvotes: 0

Views: 491

Answers (1)

Léa Gris
Léa Gris

Reputation: 19545

Populating a null delimited list of files to synchronize before running rsync with this list:

#!/usr/bin/env bash

##### Settings #####

# Location of the source data files
declare -r SRC='/local/data/'

# Destination of the data files
declare -r DEST='/import/myNas/data/'

##### End of Settings #####

set -o errexit # same as set -e, exit if command fail

declare -- _temp_fileslist

trap 'rm -f "$_temp_fileslist"' EXIT

_temp_fileslist=$(mktemp) && typeset -r _temp_fileslist

# Populate files list as null delimited entries
find "$SRC" \
  -path '*/database/session_*.db' \
  -and -not -path '*/database/session*' \
  -fprinf "$_temp_fileslist" '%P\0'

# --from0 tells rsync to read a null delimited list
# --files-from= tells to read the include list from this file
if rsync --info=progress2 --recursive \
  --from0 "--files-from=$_temp_fileslist" -- "$SRC" "$DEST";
then rsync --info=progress2 --recursive \
    --from0 "--files-from=$_temp_fileslist" \
    --checksum --remove-source-files -- "$SRC" "$DEST"
fi

Upvotes: 1

Related Questions