shell - faster alternative to "find"

Question

I'm writing a shell script wich should output the oldest file in a directory. This directory is on a remote server and has (worst case) between 1000 and 1500 (temporary) files in it. I have no access to the server and I have no influence on how the files are stored. The server is connect through a stable but not very fast line.

The result of my script is passed to a monitoring system wich in turn allerts the staff if there are too many (=unprocessed) files in the directory.

Unfortunately the monitoring system only allows a maximun execution time of 30 seconds for my script before a timeout occurs.

This wasn't a problem when testing with small directories, this wasn't a problem. Testing with the target directory over the remote-mounted directory (approx 1000 files) it is.

So I'm looking for the fastest way to get things like "the oldest / newest / largest / smallest" file in a directory (not recursive) without using 'find' or sorting the output of 'ls'.

Currently I'm using this statement in my sh script:

old)
  # return oldest file (age in seconds)
  oldest=`find $2 -maxdepth 1 -type f | xargs ls -tr | head -1`
  timestamp=`stat -f %B $oldest`
  curdate=`date +%s`
  echo `expr $(($curdate-$timestamp))`
  ;;

and I tried this one:

gfind /livedrive/669/iwt.save  -type f -printf "%T@ %P\n" | sort -nr | tail -1 | cut -d' ' -f 2-

wich are two of many variants of statements one can find using google.

Additional information: I'writing this on a FreeBSD Box with sh und bash installed. I have full access to the box and can install programs if needed. For reference: gfind is the GNU-"find" utuility as known from linux as FreeBSD has another "find" installed by default.

any help is appreciated

with kind regards, dura-zell

Answer 1

Can you try creating a shell script that will reside in the remote host and when executed will provide the required output. Then from your local machine just use ssh or something like that to run that. In this way the script will run locally there. Just a thought :-)

Answer 2

In general, you shouldn't be parsing the output of ls. In this case, it's just acting as a wrapper around stat anyway, so you may as well just call stat on each file, and use sort to get the oldest.

old) now=$(date +%s)
     read name timestamp < <(stat -f "%N %B" "$2"/* | sort -k2,2n)
     echo $(( $now - $timestamp ))

The above is concise, but doesn't distinguish between regular files and directories in the glob. If that is necessary, stick with find, but use a different form of -exec to minimize the number of calls to stat:

old ) now=$(date +%s)
      read name timestamp < <(find "$2" -maxdepth 1 -type f -exec stat -f "%N %B" '{}' + | sort -k2,2n)
     echo $(( $now - $timestamp ))

(Neither approach works if a filename contains a newline, although since you aren't using the filename in your example anyway, you can avoid that problem by dropping %N from the format and just sorting the timestamps numerically. For example:

read timestamp < <(stat -f %B "$2"/* | sort -n)
# or
read timestamp < <(find "$2" -maxdepth 1 -type f -exec stat -f %B '{}' + | sort -n)

)

Answer 3

For the oldest/newest file issue, you can use -t option to ls which sorts the output using the time modified.

-t Sort by descending time modified (most recently modified first). If two files have the same modification timestamp, sort their names in ascending lexicographical order. The -r option reverses both of these sort orders.

For the size issue, you can use -S to sort file by size.

-S Sort by size (largest file first) before sorting the operands in lexicographical order.

Notice that for both cases, -r will reverse the order of the output.

-r Reverse the order of the sort.

Those options are available on FreeBSD and Linux; and must be pretty common in most implementations of ls.

Let use know if it's fast enough.