Python script using Linux commands, does not run on Windows. Is there any way to make it OS agnostic?

Question

I have a question about a simple script. The goal of the script is to find all tgz files in a subdirectory and extract all the TIF files from them that have B4 or B5 at the end of the file name. After that it shoyld move those files to a specified subdirectory (band4 or band5, respectively).

It seems to be working fine on my Ubuntu 12.04 machine, but when a buddy of mine executes it on his Windows 7 machine it breaks. From what I understand, the script calls on linux commands and Windows cant interpret them correct (cant find *.tgz files). I was wondering if there is way to make it OS agnostic achieving the same results.

import subprocess, shlex, os, sys
cmd1 = "find . -name *.tgz" 
cmd2 = "xargs -i pigz -dv {}"
args1 = shlex.split(cmd1)
args2 = shlex.split(cmd2)

p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

cmd1 = "find . -name *.tar" 
cmd2 = "xargs -i tar -xfv {} --wildcards '*B5.TIF' '*B6.TIF' '*B8.TIF' -C %s" % repo 
args1 = shlex.split(cmd1)
args2 = shlex.split(cmd2)

p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

pathname = os.path.dirname(sys.argv[0])
b4 = os.path.abspath(pathname)+'/Band_4'
b5 = os.path.abspath(pathname)+'/Band_5'

os.mkdir(b4)
os.mkdir(b5)

cmd1 = "find . -name *B4.TIF" 
cmd2 = "xargs -i mv -if {} Band_4" 
args1 = shlex.split(cmd1)
args2 = shlex.split(cmd2)

p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

cmd1 = "find . -name *B5.TIF" 
cmd2 = "xargs -i mv -if {} Band_5" 
args1 = shlex.split(cmd1)
args2 = shlex.split(cmd2)

p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

Error Traceback:

Traceback (most recent call last):
 File "[PATH]\bands.py", line 36, in <module>
   p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE)
 File "C:\Python27\ArcGIS10.2\lib\subprocess.py", line 711, in _init_
   errread, errwrite)
 File "C:\Python27\ArcGIS10.2\lib\subprocess.py", line 948, in _execute_child
   startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
File not found - *.tgz

Answer 1

Shouldn't be too hard, as long as you understand what the script does and what the used commands / utilities do:

• find could be replaced with a combination of os.walk and fnmatch.filter (see this answer for an example)
• xargs would need to be replaced with a Python for loop that operates on the paths that were found by os.walk
• pigz could be replaced with the Python gzip module
• tar can be replaced with the tarfile library
• mv can be replaced with shutil.move

---

Here's a start:

import errno
import fnmatch
import os
import re
import shutil
import tarfile


cwd = os.getcwd()
REPO = os.path.join(cwd, "repo")


def find(directory, pattern):
    for root, dirnames, filenames in os.walk(directory):
        for fn in filenames:
            if fnmatch.fnmatch(fn.lower(), pattern.lower()):
                yield os.path.join(root, fn)


def mkdir_p(path):
    try:
        os.makedirs(path)
    except OSError as exc:
        if exc.errno == errno.EEXIST and os.path.isdir(path):
            pass
        else:
            raise


def mv(src, dst):
    try:
        shutil.move(src, dst)
    except shutil.Error, e:
        print "%s, moving on" % e


def matching_tif_files(members):
    pattern = re.compile(r'.*B[4568].tif', re.I)
    for tarinfo in members:
        if pattern.match(tarinfo.name):
            print "Extracting %s" % tarinfo.name
            yield tarinfo


targz_files = find(cwd, '*.tgz')

for tgz in targz_files:
    print tgz
    with tarfile.open(tgz) as tar:
        tar.extractall(path=REPO, members=matching_tif_files(tar))


b4 = os.path.join(cwd, 'Band_4')
b5 = os.path.join(cwd, 'Band_5')

mkdir_p(b4)
mkdir_p(b5)

b4_tifs = find(cwd, '*B4.tif')
for tif in b4_tifs:
    mv(tif, b4)

b5_tifs = find(cwd, '*B5.tif')
for tif in b5_tifs:
    mv(tif, b5)

The script actually did a couple things differently from how you described them. For example, --wildcards '*B5.TIF' '*B6.TIF' '*B8.TIF doesn't match *B4.TIF. I adapted those as I saw fit. It's certainly not perfect yet, but it should get you started.

Answer 2

Well python isn't installed on windows by default. It is possible in PHP to determine the operating system. Then based on that have different commands depending on it.

In Windows a lot can be done with vbscript, and newer windows versions have powershell, place something in the location where windows has the dir.exe commando to make it work. Its not a beautiful solution, but then you dont add much data to your friends server.

Probably asking your friend to add python to his php server would be easiest for you. For him that might be a bit complex since it needs to integrated with his php server and be executed (and not open python editor), i got blender installed but (installs python too), but .py files are not executed unless i call them like (c:\dev folder\python.exe myscript.py).

Best would be to realy install python on his windows based php server http://blog.chomperstomp.com/installing-python-mod_python-on-xampp/

Answer 3

Python itself offers platform agnostic libraries (such as shutil) to do general file system operations. There is tarfile handle TARs (w/ GZ compression). Put together, you can formulate all the script does in pure Python. It is going to be plenty of work though, so whether it is worth it as compared to using Cygwin (as suggested by @joews) greatly depends on how important the script is. A python version will be more maintainable in future.