CODEWITHSUNDEEP
linuxbashparallel-processingimagemagickxargs
Miles
Miles

Reputation: 155

How to batch convert 1000's of images across multiple sub directories in parallel using Image Magic

I have ~100 subdirectories each having ~1000 files I want to convert JPG to PNG using Image Magick under BASH for Win10 i.e. LINUX script. My script is slow, can I speed it up?

find . -type f -name '*.jpg' -exec sh -c '
    orgfile="$0"
    newfile="$(echo "$0" | sed 's/.jpg/.png/')"
    echo $orgfile $newfile
    convert $orgfile -unsharp 0x5 $newfile
    rm $orgfile
' {} \;

I like the loop process because the convert is the first in a number of processes so the input and output names can be reused. However its slow and echo is there for feedback (change to per dir?)

In a related post the following solution is given

# Runs these conversions serially
ls *.NEF | sed 's#.NEF##' | xargs -I^ convert ^.NEF ^.jpg
# Runs these conversions with 8 different processes
ls *.NEF | sed 's#.NEF##' | xargs -P8 -I^ convert ^.NEF ^.jpg

But another post warns that parallel processing may slow down a system

/media/ramdisk/img$ time for f in *.bmp; do echo $f ${f%bmp}png; done | xargs -n 2 -P 2 convert -auto-level

I think I'm getting lost in both the advanced BASH scripting and parallel processing and I have no idea of xargs.

BTW running serially is using about 25% of PC resources

Upvotes: 5

Views: 3847

Answers (4)

Ole Tange
Ole Tange

Reputation: 33748

If mogrify only uses 1 CPU you can parallelize using GNU Parallel:

parallel mogrify -unsharp 0x5 -format png ::: *.jpg

Or if the file list is too long for the shell:

ls | parallel mogrify -unsharp 0x5 -format png {}

Multiple subdirs:

find subdir1 subdir2 -name '*.jpg' | parallel mogrify -unsharp 0x5 -format png {}

Upvotes: 6

rostok
rostok

Reputation: 2137

I have slightly different approach. Instead of using xargs I gather all files that need to be processed in single text file. Shuffle it (split.exe) to distribute work equally, split it to 8 equal parts as I have 8 core cpu (shuf.exe) and run it in parallel. Here's windows batch for this:

dir /s/b *.jpg > allfiles
shuf allfiles -o allfiles
split -n l/8 allfiles
for %%i in (xaa xab xac xad xae xaf xag xah) do (
    start /separate /low /min magick mogrify -verbose -format png @%%i 
)
:loop
tasklist.exe | grep -i "magick.exe\|mogrify.exe" > nul
echo %time% %errorlevel% processing...
if %errorlevel%==0 goto loop

del xa?

There's also a loop that checks if everything's done. While split and grep can be provided by unxUtils shuf is avaiable in cygwin only. I skipped part that removes old files but this should help.

Upvotes: 0

Miles
Miles

Reputation: 155

Sorry Igor, newbie at the site couldn't post a formatted comment to your post.

Mogrify is slightly slower than xargs but a lot easier to type. Task manager showed higher disk utilisation with Mogrify and higher CPU but "spikier" (100% dropping down, xargs was consistent about 50%)

My conclusion, if it's only a number of files use Mogrify. If it's going to run all night use xargs for a cooler PC.

My time test was for one subdirectory.

find . -type d -exec sh -c '
   subdir="$0"
   echo $subdir

   #test mogrify
   time mogrify -unsharp 0x5 -format png "${subdir}/*.jpg"
   #1011.65user 30.07system 7:47.85elapsed 222%CPU

   for f in "${subdir}/*.jpg"; do echo $f ${f%jpg}png; done| time xargs -n 2 -P 8 -i convert ifile/ '{}' -unsharp 0x5 ofile/ '{}'
   #991.95user 29.35system 7:22.46elapsed 230%CPU
' {} \;

Upvotes: 0

Ipor Sircer
Ipor Sircer

Reputation: 3141

Use imagemagick's inline batch, called mogrify

mogrify -unsharp 0x5 -format png *.jpg

You can't write faster in shell. And for recursive convert use bash globbing feature:

shopt -s globstar
mogrify -unsharp 0x5 -format png **/*.jpg

Upvotes: 2

Related Questions