Bash script loop through netCDF files via placeholder and use cdo commands

Question

I want to loop through 40 netCDF files. There are 20 files with the variable PRECC and 20 files with the variable PRECL("modelmember001.PRECC.192001-200512.nc", "modelmember002.PRECC.192001-200512.nc", ... ,"modelmember020.PRECC.192001-200512.nc" and for PRECL respectively).

I need to perform multiple cdo (climate data operator) commands with the loop (add PRECC and PRECL files, and change time series form 1920-2005 to 1955-2005).

This is the code that I use:

  datadir="path_to_mydatat"
    workdir="path_to_folder_for_newfiles"
    members="{001 .. 020}"

    for model in $members
    do

    echo 'working with model' ${model}

    echo cdo -s add ${datadir}/modelmember${members}.PRECC.192001-200512.nc${datadir}/modelmember${members}.PRECL.192001-200512.nc ${workdir}/modelmember${members}PRECT.192001-200512.nc
    # echo cdo -s selyear,1955/2005 ${workdir}/modelmember${members}.PRECT.192001-200512.nc ${workdir}/modelmember${members}.PRECT.195501-200512.nc

Eventually I need 20 files with the name

"modelmember001.PRECT.195501-200512.nc", "modelmember002.PRECT.195501-200512.nc", ... , "modelmember020.PRECT.195501-200512.nc"

This is what I get when I run my code (deliberately with an "echo" in front of the cdo line):

$./cdo_add.sh

{001 .. 020}

working with model {001

cdo -s add /path_to_mydatat/modelmember{001 .. 020}.PRECC.192001-200512.nc         /path_to_mydatat/modelmember{001 .. 020}.PRECL.192001-200512.nc  /path_to_folder_for_newfiles/modelmember{001 .. 020}.PRECT.192001-200512.nc

working with model ..

cdo -s add /path_to_mydatat/modelmember{001 .. 020}.PRECC.192001-200512.nc /path_to_mydatat/modelmember{001 .. 020}.PRECL.192001-200512.nc /path_to_folder_for_newfiles/modelmember{001 .. 020}.PRECT.192001-200512.nc

working with model 020}

cdo -s add /path_to_mydatat/modelmember{001 .. 020}.PRECC.192001-200512.nc /path_to_mydatat/modelmember{001 .. 020}.PRECL.192001-200512.nc /path_to_folder_for_newfiles/modelmember{001 .. 020}.PRECT.192001-200512.nc

My code doesn't seem to loop through the members. There is something wrong with the way I use the placeholder "members" but I can't figure out how to fix it. Does anyone have a suggestion? Cheers!

Answer 1

Your code does not seem to loop because you cannot assign a brace expansion to a variable and expect it to expand when substituted in a for loop. The following saves the literal string "{001 .. 020}" to the variable members, e.g.

members="{001 .. 020}"

When you use members in for model in $members, normal word-splitting occurs because it is just a string and you loop once with 001, then with .. and finally with 020 -- not the expected sequence from 001, 002, 003, ... 020. (there should be no spaces between the number and .. to begin with -- but that still doesn't allow you to use the expansion in a variable)

To properly use the expansion, get rid of the members variable altogether and use {001..020} in the loop, e.g.

for model in {001..020}   ## (notice NO space between 001 and ..)

example:

$ for m in {001..020}; do echo $m; done
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020

That will allow you to loop with your sequence in model.

From the conversation in the comments, I know understand that you have 40 files prefixed by modelmemberXXX (where XXX is 001-020) followed by .PRECC* or .PRECL* (20 files each) that you want to coordinate feeding matching pairs to a cdo command. While the preferred way would be to loop over one matching glob, e.g. for i in modelmember*.PRECC*; do, you can also use your brace expansion approach, e.g.

for i in {001..020}
do 
    a=$(echo modelmember${i}.PRECC*)
    b=$(echo modelmember${i}.PRECL*)
    if [ -e "$a" ] && [ -f "$b" ]
    then
        printf "%s\n%s\n\n" "$a" "$b"
    fi
done

(note the [ -e "$a" ] && [ -f "$b" ] test just makes sure both files in the pair exist before proceeding with the command (printf here))

Example Output

modelmember001.PRECC.192001-200512.nc
modelmember001.PRECL.192001-200512.nc

modelmember002.PRECC.192001-200512.nc
modelmember002.PRECL.192001-200512.nc

modelmember003.PRECC.192001-200512.nc
modelmember003.PRECL.192001-200512.nc
...
modelmember020.PRECC.192001-200512.nc
modelmember020.PRECL.192001-200512.nc

You simply need to make use of $a and $b with whatever cdo_cmd you need within the loop. (as noted in the comments, you need to change to the directory containing the files, or precede the filenames with path/to/the/files)

Preferred Way

Rather than using your brace expansion, it is probably preferred to loop over one set (either PRECC or PRECL), validate the other exists, then execute the command, e.g.

for i in modelmember*.PRECC* 
do
    b="${i/PRECC/PRECL}"
    if [ -e "$i" ] && [ -f "$b" ]
    then
        printf "%s\n%s\n\n" "$i" "$b"
    fi
done

(same output)