unix ftp script to get latest file from server

Question

I have a unix script to get files via ftp looks something like this:

#!/bin/sh
HOST='1.1.1.1'
USER='user'
PASSWD='pass'
FILE='1234'

ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
cd .LogbookPlus
get $FILE
quit
END_SCRIPT
exit 0

Instead of getting a specific file, I want to get the last modified file in a folder, or all files created in the last 24 hours. Is this possible via ftp?

Answer 1

You should have definitely given some more information about the systems you are using, e.g. not every ftp server supports ls -t that @JesseParker uses. I used the opportunity and put some ideas that I have used myself for some time into a script that uses awk to to the dirty deeds. As you can see, knowing what flavor of unix your client uses would be beneficial. I have tested this script to run under Debian Wheezy GNU/Linux and FreeBSD 9.2.

#!/bin/sh


# usage: <this_script> <num_files> <date...> [ <...of...> <...max....> <...age...> ... ]
#
# Fetches files from preconfigured ftp server to current directory.
# Maximum number of files is <num_files>
# Only files that have a newer modification time than given date are considered.
# This date is given according to the local 'date' command,  which is very different
# on BSD and GNU systems, e.g.:
# 
# GNU:
#   yesterday
#   last year
#   Jan 01 1970
#
# BSD:
#   -v-1d                        # yesterday (now minus 1 day)
#   -v-1y                        # last year (now minus 1 year)
#   -f %b %e %C%y Jan 01 1970    # format: month day century year
#
# Script tries to autodetect date system, YMMV.
#
# BUGS:
#   Does not like quotation marks (") in file names,  maybe much more.
#
#   Should not have credentials inside this file,  but maybe have them
#   in '.netrc' and not use 'ftp -n'.
#
#   Plenty more.
#

HOST='1.1.1.1'
USER='user'
PASSWD='pass'
DIR='.LogbookPlus'


# Date format for numerical comparison.  Can be simply +%s if supported.
DATE_FMT=+%C%y%m%d%H%M%S


# The server's locale for date strings.
LC_SRV_DATE=C

# The 'date' command from BSD systems and that from the GNU coreutils
# are completely different.  Test for the appropriate system here:

if LC_ALL=C date -j -f "%b %e %C%y" "Jan 01 1970" $DATE_FMT > /dev/null 2>&1 ; then
    SYS_TYPE=BSDish
elif LC_ALL=C date -d "Jan 01 1970" $DATE_FMT > /dev/null 2>&1 ; then
    SYS_TYPE=GNUish
else
    echo "sh: don't know how to date ;-) sorry!"
    exit 1;
fi

# Max. number of files to get (newest files first)
MAX_NUM=$(( ${1:-1} + 0 ))  # ensure argv[1] is treated as a number
shift

# Max. age of files.  Only files newer that this will be considered.
if [ GNUish = "$SYS_TYPE" ] ; then
    MAX_AGE=$( date "$DATE_FMT" -d "${*:-yesterday}" )
elif [ BSDish = "$SYS_TYPE" ] ; then
    MAX_AGE=$( date -j "${*:--v-1d}" "$DATE_FMT" )
fi


# create temporary file
TMP_FILE=$(mktemp)
trap 'rm -f "$TMP_FILE"' EXIT INT TERM HUP


ftp -i -n $HOST <<END_FTP_SCRIPT | \
awk -v max_age="$MAX_AGE" \
    -v max_num="$MAX_NUM" \
    -v date_fmt="$DATE_FMT" \
    -v date_loc="$LC_SRV_DATE" \
    -v sys_type="$SYS_TYPE" \
    -v tmp_file="$TMP_FILE" '
BEGIN {
    # columns in the 'dir' output from the ftp server:
    # drwx------    1 user     group        4096 Apr  8  2009 Mail
    # -rw-------    1 user     group       13052 Nov 20 02:07 .bash_history
    perm=1; links=2; user=3; group=4; size=5; month=6; day=7; yeartime=8; # name=$9..$NF

    if ( "BSDish" == sys_type ) {
        date_cmd="LC_ALL=" date_loc " date -j -f"
    } else if ( "GNUish" == sys_type ) {
        date_cmd="LC_ALL=" date_loc " date -d"
    } else {
        print "awk: don'\''t know how to date ;-) sorry!" > "/dev/stderr"
        exit 1;
    }

    files[""] = ""
    file_cnt = 0
    out_cmd = "sort -rn | head -n " max_num " > " tmp_file
}

$perm ~ /^[^-]/ {   # skip non-regular files
    next
}

{
    if ( "BSDish" == sys_type ) {
        if ( $yeartime ~ /[0-9][0-9][0-9][0-9]/ ) {
            ts_fmt = "\"%b %e %C%y\"" 
        } else if ( $yeartime ~ /[0-9][0-9:[0-9][0-9]/ ) {
            ts_fmt = "\"%b %e %H:%M\"" 
        } else {
            print "has neither year nor time: " $8
            exit 1
        }
    } else { # tested in BEGIN: must be "GNUish"
        ts_fmt = ""
    }
    cmd = date_cmd " " ts_fmt " \"" $month " " $day " " $yeartime "\" " date_fmt
    cmd | getline timestamp
    close( cmd )
    if ( timestamp > max_age ) {
        # clear everything but the file name
        $perm=$links=$user=$group=$size=$month=$day=$yeartime=""
        files[ file_cnt,"name" ] = $0
        files[ file_cnt,"time" ] = timestamp
        ++file_cnt
    }
}

END {
    for( i=0; i<file_cnt; ++i ) {
        print files[ i,"time" ] "\t" files[ i,"name" ] \
        | out_cmd
    }
    close( out_cmd )

    print "quote USER '$USER'\nquote PASS '$PASSWD'\ncd \"'$DIR'\""
    i = 0
    while( (getline < tmp_file) > 0 ) {
        $1 = "" # drop timestamp
        gsub( /^ /,"" ) # strip leading space
        print "get \"" $0 "\""
    }
    print "quit"
}
' \
| ftp -v -i -n $HOST
quote USER $USER
quote PASS $PASSWD
cd "$DIR"
dir .
quit
END_FTP_SCRIPT

Answer 2

This is really pushing the FTP client further than it should be pushed, but it is possible.

Note that the LS_FILE_OFFSET might be different on your system and this won't work at all if the offset is wrong.

#!/bin/sh

HOST='1.1.1.1'
USER='user'
PASSWD='pass'
DIRECTORY='.LogbookPlus'
FILES_TO_GET=1
LS_FILE_OFFSET=57 # Check directory_listing to see where filename begins

rm -f directory_listing

# get listing from directory sorted by modification date
ftp -n $HOST > directory_listing <<fin 
quote USER $USER
quote PASS $PASSWD
cd $DIRECTORY
ls -t
quit
fin

# parse the filenames from the directory listing
files_to_get=`cut -c $LS_FILE_OFFSET- < directory_listing | head -$FILES_TO_GET`

# make a set of get commands from the filename(s)
cmd=""
for f in $files_to_get; do
cmd="${cmd}get $f
"
done

# go back and get the file(s)
ftp -n $HOST <<fin 
quote USER $USER
quote PASS $PASSWD
cd $DIRECTORY
$cmd
quit
fin

exit 0