Python: read output script twice and write two columns to csv

Question

This is my first python (2.4) program and I could use your help. I have a csv file

HOST, PATH
server1, /path/to/file1.py
server2, /path/to/file2.py
server3, /path/to/file3.py

That executes a command to each PATH on column 2/row[1]

In the output of the command I search for text and append them to new columns.

Example command output:

Command: python /path/to/file1.py
Output: server1 (NTFS) Reply:Yes

I would like make my new csv file like:

HOST, PATH, PLATFORM, REPLY
server1, /path/to/file1.py, Windows, Yes
server2, /path/to/file2.py, Linux, Yes
server3, /path/to/file3.py, BSD, No

I havnt yet got it to work with reading the same output line and appending two columns with different results. I have tried closing the writer and opening a new one within the same reader with no joy. I have tried indenting with no luck. The closest I have gotten is having it accurately search for both strings but it only writes to one column.

I have also tried first searching for the os platform and writing the changes to a new file Then, opening the new file to write another row

Of course I could force this to work by running the commands again and searching separately, but thats redundant and unnecessary.

import datetime
import csv
import os, time
from stat import * # ST_SIZE etc
from subprocess import Popen, PIPE, STDOUT

# Set Date
now = datetime.datetime.now()
today = now.strftime("%m-%d-%Y")

# Files
filename = "my_list.csv"
results = "results/results_" + today + ".csv" # Eg. results_04-14-2012.csv

# Commands
command = "python"
SP = " "

incsv = open(filename, 'rb')
try:
    reader = csv.reader(incsv)

    outcsv = open(results, 'w')
    try:
        writer = csv.writer(outcsv)

        for row in reader:
            p = Popen(command + SP + row[1], shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
            stdout, empty = p.communicate()

            print 'Command: %s\nOutput: %s\n' % (command + SP + row[1], stdout)

            if not stdout:
                writer.writerow(row + ['PLATFORM']) # Create header for new column when first line is empty on stdout
            elif 'BSD' in stdout:
                writer.writerow(row + ['BSD'])
            elif 'Linux' in stdout or 'swap' in stdout or 'LLVM' in stdout or 'EBR' in stdout:
                writer.writerow(row + ['Linux'])
            elif 'NTFS' in stdout:
                writer.writerow(row + ['Windows'])
            else:
                writer.writerow(row + ['Error Scanning'])

                    reply = open(results, 'w')
                    try:
                        writer = csv.writer(platform)

                        for row in reader:

                            if not stdout:
                                writer.writerow(row + ['REPLY']) # Create header for new column when first line is empty on stdout
                            elif 'Reply:Yes' in stdout:
                                writer.writerow(row + ['Yes'])
                            elif 'Reply:No' in stdout :
                                writer.writerow(row + ['No'])

                            else:
                                writer.writerow(row + ['Error'])

                    finally:
                        reply.close()
    finally:
        outcsv.close()
finally:
    incsv.close()

Answer 1

Collect all the information you want from the output, and write it out using a single call to writerow(). E.g.,

newdata = ['BSD', 'Yes']
...
writer.writerow(row + newdata)

Of course you will be building up newdata in steps, by parsing the script results.

PS. Update your python! 2.4 is way too old. Updating will make your life easier in countless ways.

Answer 2

You may not need csv if you have a 1:1 mapping between input lines and output lines. Just process each line and spit out a new one.

Method 1: Pipes

import sys
import subprocess

def parse_csv(input, output):
    # echo headings with new columns
    print >> output, input.readline() + ', PLATFORM, REPLY'

    for line in input.readlines():
        server, _, path = line.partition(',')
        path = path.strip()
        p = subprocess.Popen(command + ' ' + path,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
                             shell=True)
        stdout, _ = p.communicate()

        # your additional logic goes here based on contents of stdout

        # when ready to output, just print, e.g.
        print >> output, '%s, %s, %s' % (line, stdout, 'Yes')

if __name__ == '__main__':
    parse_csv(sys.stdin, sys.stdout)

Execute with

$ cat servers_and_paths.csv | python add_two_columns.py > servers_paths_and_flags.csv

Method 2: Files

If you don't want to use pipes, use with clauses to handle opening/closing:

from contextlib import closing

if __name__ == '__main__':
    with closing(open('servers_and_paths.csv')) as input,
         closing(open('servers_paths_and_flags.csv', 'wb')) as output:
        func_name(input, output)