CODEWITHSUNDEEP
bashunixcommand-line-interfacefile-manipulation
GaurangiS
GaurangiS

Reputation: 85

Recursively loop through files in bash and manipulate each file through a command

I want to recursively loop through each file in a folder from bash and do some sort of SIMPLE manipulation to them. For example, change permission, change timestamp, resize image with ImageMagick, etc., you get the idea.

I know (like most beginners) on how to do it in a directory, but recursively... ?

$ for f in *.jpg ; do convert $f -scale 25% resized/$f ; done

Let's just keep it simple. say,

$ for f in *; do touch $f; done

Upvotes: 5

Views: 6640

Answers (2)

Vitali
Vitali

Reputation: 3695

When you say bash do you really mean within bash or just scripting using common command-line tools?

Scripting tools together

find . -type f -exec chmod u+x {} \;

What this means is for every file in the current directory make it executable. The \; is passing the ; to the find command which interprets it as "invoke the exec string on each found path individually". You can replace \; with \+ which tells find to first gather all paths & substitute it for {} all at once. Generally \+ can be more efficient but you have to be careful with command-line lengths as there are limits. What you can do then is combine it with xargs:

find . -type f -print0 | xargs -0 -P $(nproc) -I{} chmod u+x {}

What this does is it tells find to use the null character as a terminator instead of newlines. This ensures that you process each entry correct even if it has arbitrary spaces or random UTF characters (\0 is not a valid part of a path). The -0 option to xargs tells it to use \0 as the separator when reading arguments instead of newliens. The -P option says to run the command in parallel N times where in this case N is the output of the nproc command which prints the number of processors. -I is the substitution string and the rest is the command string to process.

The man pages for find & xargs are good to explore.

Natively within Bash

On the off-chance you're looking for a solution wholly within Bash & no external tools, it's a bit more complicated & would involve some more advanced Bash-specific language constructs where you implement find yourself. To iterate over the contents of a directory, you'd do something like for path in <dir>; do. Then you'd use the test built-in [[ -d "$path" ]] to determine if it's a directory, [[ -f "$path" ]] if it's a file etc (man test has many of the explanations but note that's the standalone test executable which has subtle differences & pitfalls from the more feature-filled & safer bash version [[ ]].

Working with bash arrays: https://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_10_02.html Bash test introduction: https://www.tldp.org/LDP/abs/html/testconstructs.html

What that test introduction doesn't mention is things like regular expressions which would be part of that syntax. Bash also has powerful options for manipulating the contents of variables: https://www.tldp.org/LDP/abs/html/parameter-substitution.html

In practice though, anything even moderately complex (whether within bash or by combining tools), is probably better maintained & easier to read in Python (speaking as someone with lots and lots of extensive experience in Bash).

find_files() {
  if [[ ! -x "$1" ]]; then
     echo "$1 isn't a directory" >&2
     return 1
  fi

  local dirs=("$1")

  while [[ "${#dirs[@]}" -gt 0 ]]; do
    local dir="${dirs[0]}"
    dirs=("${dirs[@]:1}") # pop the element from above

    # whitespace in filenames iterated will be a problem. Look to the IFS
    # variable to handle those more gracefully.
    for p in "${dir}"/*; do 
      if [[ -d "$p" ]]; then
         dirs+=("$p")
      elif [[ -f "$p" ]]; then
         echo "$p"
      fi
    done
  done
}

for f in $(find_files .); do
    chmod u+x "$f"
done

As you can see this is more complicated, trickier, & slower than just using the find/xargs binaries. You would never want to write something like that in reality. You could even get fancier where you convert find_files into process by having it take a command that you then evaluate as you're iterating (instead of echo'ing the path) via eval. eval is super tricky & can be a security exploit.

Upvotes: 4

wjandrea
wjandrea

Reputation: 33135

Use globstar:

shopt -s globstar
for f in ./**; do
    touch "$f"
done

From the Bash manual:

globstar

If set, the pattern ‘**’ used in a filename expansion context will match all files and zero or more directories and subdirectories. If the pattern is followed by a ‘/’, only directories and subdirectories match.

BTW, always quote your variables, and use ./ in case of filenames that look like options.

This is based on codeforester's answer here

Upvotes: 11

Related Questions