Pipe subprocess to a file in Linux

Question

I'm beating my head against a wall trying to get my code to work properly. In essence, I've written a python script that will be called by cron. The script is to run once per week to perform database maintenance. My python script will call a perl script that performs the actual maintenance. I've had to build it this way because the perl script will be run a total of 9 separate times. There are 3 different categories of DB items and each category requires 3 different actions from the perl script. If a single action in any category fails, the script is to break from that category and move to the next one (each action is dependent upon the previous action).

My script is failing to execute properly and I'm not sure why. I'm trying to troubleshoot it but my logging logic isn't working properly either. The only output I get in my log file is the simple "print" commands that I've written into my script for notation and formatting.

I can run the perl script just fine from the CLI directly, but it fails to properly run when invoked from my python script. Can anyone point me in the right direction please?

EDIT: In response to a question, I don't have a good explanation of how the perl script is failing. I'm not getting the stdout or stderr from the perl script into my log file so I don't yet know why it's failing. I didn't create the perl script, it was provided with the software suite that I'm administering.

Here's the log file output:

------------------------------------------------------------------------------------------------------
Weekly Voyence archival, deletion, and purging of jobs, events, and revisions beginning at 14:46:11
Block failed, continuing to next block
------
Block failed, continuing to next block
------
Block failed, continuing to next block
------
Weekly Voyence archival, deletion, and purging of jobs, events, and revisions complete at 14:46:11

Here is output from running the perl script from the CLI (without errors):

Script started at Mon Nov  7 14:59:53 2016

Archving/Pruning 0 Parent jobs older than 90 days...
Archving/Pruning 0 Child Jobs older than 90 days...

No Records avaialble for archive..jobs Exiting
Script Completed.

The database utility console log can be found at /opt/voyence/tools/db-   utility/logs/dbutil_console.log

Script exited at Mon Nov  7 14:59:58 2016

Here is the python script:

#! /usr/bin/python
import sys
import time
import os
import subprocess
def SubProcessCaller(PATH,CallLoad,LogObj,LOGFILE):
    try:
        # subprocess.Popen(["/usr/bin/perl", "-i", "/opt/voyence/tools/db-    utility/database-utility.pl", CallLoad], stderr=LogObj.write, stdout=LogObj.write, shell=true)
        concatvar = "/usr/bin/perl -I /opt/voyence/tools/db-utility/common /opt/voyence/tools/db-utility/database-utility.pl" + CallLoad + " >> " + LOGFILE
        subprocess.check_output(concatvar.split(), shell=True)
        return True
    except:
        LogObj.write("Block failed, continuing to next block" "\n")
        return False

if __name__ == "__main__":

    #DECLARE VARIABLES AND OBJECTS
    TIME = time.strftime("%H:%M:%S")
    PATH = "/opt/voyence/tools/db-utility/database-utility.pl"
    DATE = time.strftime("%Y%m%d")
    LOGFILE = "/var/log/voyence-archive/" + DATE + "voyence-archive.log"
    LOG = open(LOGFILE, "a+")
    #CREATE NEW LOG FILE
    LOG.write("------------------------------------------------------------------------------------------------------\n")
    LOG.write("Weekly Voyence archival, deletion, and purging of jobs, events, and revisions beginning at " + TIME + "\n")
    #JOB ARCHIVAL, DELETION, AND PURGING
    JobActionsDict = [" archive jobs 90"," delete jobs 90"," purge jobs 275"]
    #EVENT ARCHIVAL, DELETION, AND PURGING
    EventActionsDict = [" archive events all 90"," delete events all 90"," purge events all 275"]
    #REVISION ARCHIVAL, DELETION, AND PURGING
    RevisionActionsDict = [" archive revisions 40"," purge revisions 60","  delete revisions '3 months' 40"]

    ActionsTuple = (JobActionsDict, EventActionsDict, RevisionActionsDict)

    for dict in ActionsTuple:
        for item in dict:
            SPCBool = SubProcessCaller(PATH,item,LOG,LOGFILE)
            if SPCBool == False:
                break
        LOG.write("------\n")
    #CLOSE AFTER ALL LOGS HAVE BEEN WRITTEN
    LOG.write("Weekly Voyence archival, deletion, and purging of jobs, events, and revisions complete at " + TIME + "\n")
    LOG.close()

Answer 1

Untested, but more idiomatic, Python. The major changes are to bypass the shell altogether when calling the Perl script, and to log any errors that might be raised by subprocess.call to aid in debugging the problem. (Consider using the logging module instead, but this has enough changes already.)

#! /usr/bin/python
import sys
import time
import os
import subprocess

def subprocess_caller(path, call_load, log_obj):
    subprocess.call(["/usr/bin/perl",
                     "-I",
                     "/opt/voyence/tools/db-utility/common",
                     path,
                     call_load],
                     stdout=log_obj)

log_header = """\
------------------------------------------------------------------------------------------------------
Weekly Voyence archival, deletion, and purging of jobs, events, and revisions beginning at %s
""" %(time.strftime("%H:%M:%S"),)

if __name__ == "__main__":

    #declare variables and objects
    path = "/opt/voyence/tools/db-utility/database-utility.pl"
    logfile = time.strftime("/var/log/voyence-archive/%Y%m%dvoyence-archive.log")

    #create new log file
    with open(logfile, "a+") as log:
        log.write(hog_header)
        #job archival, deletion, and purging
        job_actions = [" archive jobs 90"," delete jobs 90"," purge jobs 275"]
        #event archival, deletion, and purging
        event_actions = [" archive events all 90"," delete events all 90"," purge events all 275"]
        #revision archival, deletion, and purging
        revision_actions = [" archive revisions 40"," purge revisions 60","  delete revisions '3 months' 40"]

        actions = job_actions + event_actions + revision_action:

        for action_type in actions:
            for item in action_type:
                try:
                    subprocess_caller(path, item, log)
                except Exception as exc:
                    log_obj.write(exc + "\n")
                    log_obj.write("Block failed, continuing to next block\n")
                    break
                log.write("------\n")

        log.write("Weekly Voyence archival, deletion, and purging of jobs, events, and revisions complete at %s\n" %
                  (time.strftime("%H:%M:%S"),))

Answer 2

concatvar = "/usr/bin/perl -I /opt/voyence/tools/db-utility/common /opt/voyence/tools/db-utility/database-utility.pl" + CallLoad + " >> " + LOGFILE

This concatenates the name of your program (database-utility.pl) with the contents of the CallLoad variable - without putting a space between them. So, unless the value in CallLoad starts with a space, you'll actually be looking for a program called database-utility.plWhateverIsInCallLoad.