Tar got into a for loop

Question

I have bunch of csv files with date in it I want to tar each and every day files into the separate tar.gz file . However when I do that . It got stuck for single day output or tar is created with single file for single day.

Answer 1

You are making this a bit harder than it needs to be. If all you want to do is tar files older than "5 days ago", you already know how to get the seconds since epoch for "5 days ago", each day is 86400 seconds, so loop subtracting i * 86400 seconds each time for a far as you want to go back to generate the date of interest and the tar filenames, e.g.

$ d=$(date -d"5 days ago" +%s)
for i in {1..30}; do 
    dt="$(date -d@$((d - i * 86400)) +%Y%m%d)"
    fn="$dt.tar.gz"
    echo "tar -czf $fn *$dt*"
done

It uses simply globbing to find all files matching the date to include in each daily tar file, e.g., the output from the snippet above is:

tar -czf 20180326.tar.gz *20180326*
tar -czf 20180325.tar.gz *20180325*
tar -czf 20180324.tar.gz *20180324*
tar -czf 20180323.tar.gz *20180323*
tar -czf 20180322.tar.gz *20180322*
tar -czf 20180321.tar.gz *20180321*
tar -czf 20180320.tar.gz *20180320*
...
tar -czf 20180304.tar.gz *20180304*
tar -czf 20180303.tar.gz *20180303*
tar -czf 20180302.tar.gz *20180302*
tar -czf 20180301.tar.gz *20180301*
tar -czf 20180228.tar.gz *20180228*
tar -czf 20180227.tar.gz *20180227*
tar -czf 20180226.tar.gz *20180226*
tar -czf 20180225.tar.gz *20180225*

You can easily adapt this logic to your function above and is much more efficient than a triple-pipe inside a command substitution just to generate the DIFF for each day.

---

Edit in Response to Comment Re:Recursive

Above was intended to provide an example of a simple way to compute the wanted dates and form the tar filename and a simple search glob to enable you to put the pieces together however you like. If you are having difficulty putting it into a form that will recursively find the file with a common date, then using find in a command substitution provides an easy way of making the search recursive using the same glob shown above.

The find command can take the form of:

tar -czf "$fn" $(find "$srchpath" -type f -name *"$dt"*) 2>/dev/null

Where find will search for files in "$srchpath" and below for files matching the "$dt" glob. If there are any missing days in the range, the "tar: Cowardly refusing to create an empty archive" error message is redirected into the bit-bucket.

A short example script taking the srchpath as the first argument (default '.') and the range of days before "5 days ago" to search (default 30) as the second argument, you could do something like the following.

#!/bin/bash

srchpath="${1:-.}"      ## search path (default .)
daterng="${2:-30}"      ## date range (default 30)

test -d "$srchpath" || {    ## validate search path
    printf "error: path not found '%s'.\n" "$srchpath"
    exit 1
}
test "$daterng" -eq "$daterng" 2>/dev/null || { ## validate range
    printf "error: non-integer range entered '%s'.\n" "$daternd"
    exit 1
}
d=$(date -d"5 days ago" +%s)    ## get 5 days ago

for ((i = 1; i <= daterng; i++)); do    ## loop over days in range
    dt="$(date -d@$((d - i * 86400)) +%Y%m%d)"  ## get date yyyymmdd
    fn="$dt.tar.gz"     ## form filename yyyymmdd.tar.gz
    echo "tar -czf $fn \$(find $srchpath -type f -name *$dt*)" ## echo command
    ## recursively compress matching files (no spaces allowed in name)
    tar -czf "$fn" $(find "$srchpath" -type f -name *"$dt"*) 2>/dev/null
done

(note: the files to compress must not have spaces in their name. If that is a possibility, use the -print0 option for find and pipe the output to xargs calling tar)