Python, stdout , C and SWIG

Question

Suppose i have a function in c:

hello.c

void hello()
{
  printf("Hello How are you");
}

This is now wrapped into a python function hello.py.

Now, when I run the python function hello.py, I get the output but I want to save it to a variable.

I tried using this method:

import sys
import StringIO

stdout = sys.stdout  
result = StringIO.StringIO()
sys.stdout = result
hello()
sys.stdout = stdout
result_osr_string = result.getvalue()
print result.getvalue()

I think the issue is with the swig to python conversion. Because when I try the above method for a simple python function, it works.

I tried using fprintf instead of printf but even that didn't work.

Answer 1

You can make a solution, exactly as specified, provided you're willing to use some platform specific functions. I'll work with GNU libc in my examples, but BSD certainly has an equivalent and there are options to emulate the same on Windows too. I will also focus on stdout and output in general, although the corresponding examples for stdin requires only tweaks.

To actually meet your requirements there are two things we need to solve, so I'll address these sequentially.

Mapping a Python IO object to FILE*:

Firstly we need to find a way to let operations on a FILE* actually be reflected on.

On GNU/Linux libc provides fopencookie as a GNU specific extension. (The BSD equivalent is funopen, Windows seems to be more complex and need a thread and anonymous pipe to emulate the same).

With fopencookie we can create a FILE* object that behaves exactly as you'd expect, but maps the underlying IO calls onto calls to function pointers. So all we need to do to make this happen is supply a few functions that use the Python C API to do the work

Note that if all the objects you cared about in Python were instances of file you this just use some file specific C API calls instead of fopencookie:

%module test

%{
//#define _GNU_SOURCE - not needed, Python already does that!
#include <stdio.h>

static ssize_t py_write(void *cookie, const char *buf, size_t size) {
  // Note we might need to acquire the GIL here, depending on what you target exactly
  PyObject *result = PyObject_CallMethodObjArgs(cookie, PyString_FromString("write"),
                                                PyString_FromStringAndSize(buf, size), NULL);

  (void)result; // Should we DECREF?
  return size; // assume OK, should really catch instead though
}

static int py_close(void *cookie) {
  Py_DECREF(cookie);
  return 0;
}

static FILE *fopen_python(PyObject *output) {
  if (PyFile_Check(output)) {
    // See notes at: https://docs.python.org/2/c-api/file.html about GIL
    return PyFile_AsFile(output);
  }

  cookie_io_functions_t funcs = {
    .write = py_write,
    .close = py_close,
  };
  Py_INCREF(output);
  return fopencookie(output, "w", funcs);
}
%}
    
%typemap(in) FILE * {
  $1 = fopen_python($input);
}

%typemap(freearg) FILE * {
  // Note GIL comment above here also
  // fileno for fopencookie always returns -1
  if (-1 == fileno($1)) fclose($1);
}

%inline %{
  void hello(FILE *out)
  {
    fprintf(out, "Hello How are you\n");
  }
%}

Which is enough to let the following Python work:

import sys
import StringIO

stdout = sys.stdout
result = StringIO.StringIO()
sys.stdout = result

from test import hello
hello(sys.stdout)

sys.stdout = stdout
result_osr_string = result.getvalue()
print "Python: %s" % result.getvalue()

By passing the FILE* in as an argument at every function call this ensures we never end up with a stale reference to a Python handle that was later replaced elsewhere.

Making the process transparent

In the above example we have to explicitly state which IO object to use for each function call. We can simplify this and move close towards your example by using an argument that get automatically filled in by the wrapper code. In this instance I'll modify the typemaps above to automatically use sys.stdout for arguments like FILE *py_stdout:

%typemap(in) FILE * (int needclose) {
  $1 = fopen_python($input);
  needclose = !PyFile_Check($input);
}

%typemap(freearg) FILE * {
   // Note GIL comment above
   if (needclose$argnum) fclose($1);
}

%typemap(in,numinputs=0) FILE *py_stdout (int needclose) {
  PyObject *sys = PyImport_ImportModule("sys");
  PyObject *f = PyObject_GetAttrString(sys, "stdout");
  needclose = !PyFile_Check(f);
  $1 = fopen_python(f);
  Py_DECREF(f);
  Py_DECREF(sys);
}


%inline %{
  void hello(FILE *py_stdout)
  {
    fprintf(py_stdout, "Hello How are you\n");
  }
%}

Notice that here the typemap for FILE *py_stdout "specialises" instead of completely replaces the generic FILE * typemap, so both variants can be used within the same interface. You can also use %apply instead of actually renaming the argument, to avoid needing to modify existing header files if you use %import.

This now means we can just call hello() in Python and have the value of sys.stdout implicitly passed into the function at every call.

We've also improved an issue in the first example I showed by properly keeping track of if we're supposed to be calling fclose on the FILE object at the end of the function call or not. This is in a variable local to the typemap that we set inside the input typemap that matched our specific case.

Actually change stdout in C

Normally if you want to really change stdout in C you would do that with freopen. The reasoning for this and not just doing an assignment is thatstdout isn't guaranteed to be a modifiable lvalue.

In practice though you used to be able to get away with this on some platforms. In my testing though Linux/GCC is no longer one of those platforms and my assignment had no effect on the behaviour.

We also can't use freopen in this situation, at least not for the case where we used fopencookie because there isn't a file path to point freopen at. For the case where the Python file object mapped neatly onto a real FILE* on Linux we could use something like the following pseudo code:

freopen("/proc/self/fd/%d" % fileno(f), "w", stdout); 

To replace stdout. We would still need to arrange for this to happen before every C call, (probably abusing the %exception mechanism to make that hook) to keep the Python->C stdout mapping current. This is pretty ugly and limited in use, as well as somewhat flawed for multithreaded applications.

Another alternative approach would be to hook modifications to sys.stdout etc. via a modified trick like this. Again that's pretty ugly and still doesn't solve the general case.

Finally if totally replacing stdout, stderr, and stdin in existing C code really is something you want to do I'd suggest the following. You spawn a thread per file handle, with a pipe() pair for each. You then use freopen to open one end of the pipe (depending on which handle it is) from /proc (or via a named pipe in Windows). The other end of each pipe then gets used in one thread to block waiting for IO to occur on the pipe. When IO occurs your code then looks up the current Python file handle and proxies the call to that handle. This is reliable, correct, portable enough and fairly straightforward.

Improvements

If you use this code for real you probably want to do the following things:

1. Address GIL issues as commented
2. Make FILE* objects able to be RW instead of just W
3. Add corresponding stderr and stdin helper typemaps
4. Provide BSD/Windows alternative code paths instead of fopencookie.

Answer 2

The c function printf doesn't pay any attention to the value stored in python's sys.stdout. You should probably use sprintf or snprintf to print your text to a cstring and return the char * from your c function. Swig will wrap this in a python string object.