CODEWITHSUNDEEP
bash
Daley
Daley

Reputation: 23

Read file by line then process as different variable

I have created a text file with a list of file names like below 

022694-39.tar
022694-39.tar.2017-05-30_13:56:33.OLD
022694-39.tar.2017-07-04_09:22:04.OLD
022739-06.tar
022867-28.tar
022867-28.tar.2018-07-18_11:59:19.OLD
022932-33.tar

I am trying to read the file line by line then strip anything after .tar with awk and use this to create a folder unless it exists.

Then the plan is to copy the original file to the new folder with the original full name stored in $LINE.

$QNAP= "Path to storage"
$LOG_DIR/$NOVA_TAR_LIST= "Path to text file containing file names"

while read -r LINE; do
    CURNT_JOB_STRIPED="$LINE | `awk -F ".tar" '{print $1}'`"
    if [ ! -d "$QNAP/$CURNT_JOB_STRIPED" ]
        then
        echo "Folder $QNAP/$CURNT_JOB_STRIPED doesn't exist."
        #mkdir "$QNAP/$CURNT_JOB_STRIPED"
    fi
done <"$LOG_DIR/$NOVA_TAR_LIST"

Unfortunately this seems to be trying to join all the file names together when trying to create the directories rather than doing them one by one and I get a File name too long

output:

......951267-21\n951267-21\n961075-07\n961148-13\n961520-20\n971333-21\n981325-22\n981325-22\n981743-40\n999111-99\n999999-04g\n999999-44': File name too long

Apologies if this is trivial, bit of a rookie...

Upvotes: 2

Views: 143

Answers (2)

danrodlor
danrodlor

Reputation: 1459

Try modifying your script as follows:

CURNT_JOB_STRIPED=$(echo "${LINE}" | awk -F ".tar" '{print $1}')

You have to use $(...) for command substitution. Also, you should print the variable LINE in order to prevent the shell from interpreting its value as a command but passing it to the next command of the pipe (as an input) instead. Finally, you should remove the backticks from the awk expression (this is the deprecated syntax for command substitution) since what you want is the result from the piping commands.

For further information, take a look over http://tldp.org/LDP/abs/html/commandsub.html

Alternatively, and far less readable (neither with a higher performance, thus just as a "curiosity"), you can just use instead of the whole while loop:

xargs -I{} bash -c 'mkdir -p "${2}/${1%.tar*}"' - '{}' "${QNAP}" < "${LOG_DIR}/${NOVA_TAR_LIST}"

Upvotes: 1

lw0v0wl
lw0v0wl

Reputation: 674

The problem is with the CURNT_JOB_STRIPED="$LINE | `awk -F ".tar" '{print $1}'`" line.

The `command` is legacy a syntax, $(command) should be used instead.

$LINE variable should be printed so awk can receive its value trough a pipe.

If you run the whole thing in a sub shell ( $(command) ) you can assign the output into a variable: var=$(date)

Is is safer to put variables into ${} so if there is surrounding text you will not get unexpected results.

This should work: CURNT_JOB_STRIPED=$(echo "${LINE}" | awk -F '.tar' '{print $1}')

With variable substitution this can be achieved with more efficient code, and it also clean to read I believe.

Variable substitution is not change the ${LINE} variable so it can be used later as the variable that have the full filename unchanged while ${LINE%.tar*} cut the last .tar text from the variable value and with * anything after that.

while read -r LINE; do
        if [ ! -d "${QNAP}/${LINE%.tar*}" ]
        then
                echo "Folder ${QNAP}/${LINE%.tar*} doesn't exist."
                #mkdir "${QNAP}/${LINE%.tar*}"
        fi
done <"${LOG_DIR}/${NOVA_TAR_LIST}"

This way you not store the directory name as variable and ${LINE} only store the filename. If You need it into a variable you can do that easily: var="${LINE%.tar*}"

Variable Substitution:

There is more i only picked this 4 for now as they similar and relevant here.

${var#pattern} - Use value of var after removing text that match pattern from the left

${var##pattern} - Same as above but remove the longest matching piece instead the shortest

${var%pattern} - Use value of var after removing text that match pattern from the right

${var%%pattern} - Same as above but remove the longest matching piece instead the shortest

Upvotes: 1

Related Questions