Redirect stdout and stderr of an external library in Python

Question

I'm trying to collect the log of the search of the Google OR-tools library. Specifically, I'm using python to write a program that uses the library to solve a TSP instance. What I'm trying to do is to make the library solve the same instance with different methods and collect the output of the searching (which the library writes to stderr). The problem is that the library was written and compiled in C++ and only the wrappers for python have been written. I tried to use redirect_stderr and also the sys module to set stderr to another file, but the result is the same: the output still gets written on stderr and shown on the console. Because I've written a multithreaded program, shell redirection is not really an option. Is there a way to solve this issue?

The script below was taken from the Google OR-Tools documentation and slightly modified to include the specification of the methods to use and my attempt to redirect the srderr file:

"""Simple travelling salesman problem between cities."""

from __future__ import print_function
from ortools.constraint_solver import routing_enums_pb2
from ortools.constraint_solver import pywrapcp
from contextlib import redirect_stderr, redirect_stdout
import sys


def create_data_model():
    """Stores the data for the problem."""
    data = {}
    data['distance_matrix'] = [
        [0, 2451, 713, 1018, 1631, 1374, 2408, 213, 2571, 875, 1420, 2145, 1972],
        [2451, 0, 1745, 1524, 831, 1240, 959, 2596, 403, 1589, 1374, 357, 579],
        [713, 1745, 0, 355, 920, 803, 1737, 851, 1858, 262, 940, 1453, 1260],
        [1018, 1524, 355, 0, 700, 862, 1395, 1123, 1584, 466, 1056, 1280, 987],
        [1631, 831, 920, 700, 0, 663, 1021, 1769, 949, 796, 879, 586, 371],
        [1374, 1240, 803, 862, 663, 0, 1681, 1551, 1765, 547, 225, 887, 999],
        [2408, 959, 1737, 1395, 1021, 1681, 0, 2493, 678, 1724, 1891, 1114, 701],
        [213, 2596, 851, 1123, 1769, 1551, 2493, 0, 2699, 1038, 1605, 2300, 2099],
        [2571, 403, 1858, 1584, 949, 1765, 678, 2699, 0, 1744, 1645, 653, 600],
        [875, 1589, 262, 466, 796, 547, 1724, 1038, 1744, 0, 679, 1272, 1162],
        [1420, 1374, 940, 1056, 879, 225, 1891, 1605, 1645, 679, 0, 1017, 1200],
        [2145, 357, 1453, 1280, 586, 887, 1114, 2300, 653, 1272, 1017, 0, 504],
        [1972, 579, 1260, 987, 371, 999, 701, 2099, 600, 1162, 1200, 504, 0],
    ]  # yapf: disable
    data['num_vehicles'] = 1
    data['depot'] = 0
    return data


def print_solution(manager, routing, solution):
    """Prints solution on console."""
    print('Objective: {} miles'.format(solution.ObjectiveValue()))
    index = routing.Start(0)
    plan_output = 'Route for vehicle 0:\n'
    route_distance = 0
    while not routing.IsEnd(index):
        plan_output += ' {} ->'.format(manager.IndexToNode(index))
        previous_index = index
        index = solution.Value(routing.NextVar(index))
        route_distance += routing.GetArcCostForVehicle(previous_index, index, 0)
    plan_output += ' {}\n'.format(manager.IndexToNode(index))
    plan_output += 'Route distance: {}miles\n'.format(route_distance)
    print(plan_output)


def main():
    """Entry point of the program."""
    # Instantiate the data problem.
    data = create_data_model()

    # Create the routing index manager.
    manager = pywrapcp.RoutingIndexManager(len(data['distance_matrix']),
                                           data['num_vehicles'], data['depot'])

    # Create Routing Model.
    routing = pywrapcp.RoutingModel(manager)


    def distance_callback(from_index, to_index):
        """Returns the distance between the two nodes."""
        # Convert from routing variable Index to distance matrix NodeIndex.
        from_node = manager.IndexToNode(from_index)
        to_node = manager.IndexToNode(to_index)
        return data['distance_matrix'][from_node][to_node]

    transit_callback_index = routing.RegisterTransitCallback(distance_callback)

    # Define cost of each arc.
    routing.SetArcCostEvaluatorOfAllVehicles(transit_callback_index)

    # Setting first solution heuristic.
    search_parameters = pywrapcp.DefaultRoutingSearchParameters()
    search_parameters.first_solution_strategy = (
        routing_enums_pb2.FirstSolutionStrategy.PATH_CHEAPEST_ARC)
    search_parameters.local_search_metaheuristic = (routing_enums_pb2.LocalSearchMetaheuristic.TABU_SEARCH)
    search_parameters.time_limit.seconds = 1
    search_parameters.log_search = True

    # This is the first approach and it does not work
    # (Yes, I'm sure that it's routing.SolveWithParameters() that logs the search
    # parameters to stderr.
    with open("output.txt", "wt") as f:
        with redirect_stderr(f):
            solution = routing.SolveWithParameters(search_parameters)

    # Print solution on console.
    if solution:
        print_solution(manager, routing, solution)


if __name__ == '__main__':
    main()

The second approach was:

#[code omitted: see above]
    search_parameters.time_limit.seconds = 1
    search_parameters.log_search = True

    # This is the second approach and it does not work
    # (Yes, I'm sure that it's routing.SolveWithParameters() that logs the search
    # parameters to stderr.
    sys.stderr = open("output.txt", "wt")
    solution = routing.SolveWithParameters(search_parameters)
#[code omitted: see above]

Answer 1

You will probably need to redirect the stdout/stderr file descriptors, not the stream objects like the contextmanager bits would do.

There's a neat article about this, but it boils down to something like

redirect_file = tempfile.TemporaryFile(mode='w+b')
stdout_fd = sys.stdout.fileno()
sys.stdout.close()
os.dup2(redirect_file.fileno(), stdout_fd)

and the same for stderr. (The linked article shows how to hold on to the original stdout/stderr too to restore previous behavior.)