python script using subprocess, redirect ALL output to file

Question

I am writing something for static analysis of source code in different languages. As anything has to be open source and callable from command line I now have downloaded one tool per language. So I decided to write a python script listing all source files in a project folder and calling the respective tool.

So part of my code looks like this:

import os
import sys
import subprocess
from subprocess import call
from pylint.lint import Run as pylint


class Analyser:

    def __init__(self, source=os.getcwd(), logfilename=None):

        # doing initialization stuff
        self.logfilename = logfilename or 'CodeAnalysisReport.log'

        self.listFiles()
        self.analyseFiles()


    def listFiles(self):
    # lists all source files in the specified directory


    def analyseFiles(self):

        self.analysePythons()
        self.analyseCpps()
        self.analyseJss()
        self.analyseJavas()
        self.analyseCs()


if __name__ == '__main__':

    Analyser()

Let's have at a look at the C++ files part (I use Cppcheck to analyse those):

    def analyseCpps(self):

        for sourcefile in self.files['.cc'] + self.files['.cpp']:
            print '\n'*2, '*'*70, '\n', sourcefile
            call(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile])

The console output for one of the files (it's just a random downloaded file) is:

**********************************************************************
C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc
Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.
(information) Cppcheck cannot find all the include files (use --check-config for details)

Line 1 and 2 coming from my script, lines 3 to 7 coming from Cppcheck.

And this is what I want to save to my log file, for all the other files too. Everything in one single file.

Of course I have searched SO and found some methods. But none is working completely.

First try:

Adding sys.stdout = open(self.logfilename, 'w') to my constructor. This makes line 1 and 2 of the above showed output be written to my log file. The rest is still shown on console.

Second try:

Additionaly, in analyseCpps I use:

call(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], stdout=sys.stdout)

This makes my log file to be:

Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...


********************************************************************** 
C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc

and the console output is:

[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.

Not what I want.

Third try:

Using Popen with pipe. sys.stdout is back to default.

As preliminary work analyseCpps now is:

for sourcefile in self.files['.cc'] + self.files['.cpp']:

    print '\n'*2, '*'*70, '\n', sourcefile
    p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=subprocess.PIPE)
    p.stdout.read()

p.stdout.read() shows only the last line of my desired output (line 7 in code box 3)

Fourth try:

Using subprocess.Popen(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], stdout=open(self.logfilename, 'a+')) just writes the one line Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc... to my logfile, the rest is shown on the console.

Fifth try:

Instead of subprocess.Popen I use os.system, so my calling command is:

os.system('C:\CodeAnalysis\cppcheck\cppcheck --enable=all %s >> %s' % (sourcefile, self.logfilename))

This results in the same log file as my fourth try. If I type the same command directly in the windows console the result is the same. So I guess it it is not exactly a python problem but still:

If it is on the console there must be a way to put it in a file. Any ideas?

---

E D I T

Foolish me. I'm still a noob so I forgot about the stderr. That's where the decisive messages are going to.

So now I have:

def analyseCpps(self):

    for sourcefile in self.files['.cc'] + self.files['.cpp']:
        p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stderr=subprocess.PIPE)
        with open(self.logfilename, 'a+') as logfile:
            logfile.write('%s\n%s\n' % ('*'*70, sourcefile))
            for line in p.stderr.readlines():
                logfile.write('%s\n' % line.strip())

and it's working fine.

---

ANOTHER EDIT

according to Didier's answer:

with sys.stdout = open(self.logfilename, 'w', 0) in my constructor:

def analyseCpps(self):

    for sourcefile in self.files['.cc'] + self.files['.cpp']:
        print '\n'*2, '*'*70, '\n', sourcefile
        p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=sys.stdout, stderr=sys.stdout)

Answer 1

There are several problems:

• you should redirect both stdout and stderr
• you should use unbuffered files if you want to mix normal print and the output of launched commands.

Something like this:

import sys, subprocess

# Note the 0 here (unbuffered file)
sys.stdout = open("mylog","w",0)

print "Hello"
print "-----"

subprocess.call(["./prog"],stdout=sys.stdout, stderr=sys.stdout)
print "-----"
subprocess.call(["./prog"],stdout=sys.stdout, stderr=sys.stdout)
print "-----"

print "End"

Answer 2

Try to redirect stdout and stderr to a logfile:

import subprocess

def analyseCpps(self):
     with open("logfile.txt", "w") as logfile:
         for sourcefile in self.files['.cc'] + self.files['.cpp']:
             print '\n'*2, '*'*70, '\n', sourcefile
             call(['C:\\CodeAnalysis\\cppcheck\\cppcheck',
                   '--enable=all', sourcefile], stdout=logfile,
                   stderr=subprocess.STDOUT)

In this example the filename is hardcoded, but you should be able to change that easily (to your self.logfilename or similar).

Answer 3

You need to redirect stderr too, you can use STDOUT or pass the file object to stderr=:

from subprocess import check_call,STDOUT
with open("log.txt","w") as f:
     for sourcefile in self.files['.cc'] + self.files['.cpp']:
        check_call(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile],
                   stdout=f, stderr=STDOUT)