rmarkdown::render() output to stdout

Question

When I render() an *.Rmd file locally in RStudio, the output from the render() function is displayed in the console:

Basic .Rmd file:

---
title: "test"
output: html_document
---

```{r setup, include=FALSE}

sink("./output.txt", type = 'output')

knitr::opts_chunk$set(echo = TRUE)
```

## Summary

```{r cars}
summary(cars)
```

## Error

```{r}

dbGetQuery()

```

To build:

library(rmarkdown)

render('./test.rmd')

Output:

This is great when I'm creating reports locally and I can see the progress and errors thrown (if any). I need to monitor this output in stdout (or stderr) but I can't sink this output to that location because knitr is using capture.input which uses sink() (see first comment). I even tried sinking to a file instead but though the file output.txt is created, there is nothing recorded in that file.

This is an issue for me because I'm using render() in a Docker container and I can't send the chunk output from the .Rmd file in the Docker container to stderr or stdout. I need to monitor the chunk output for errors in the container R code inside the .Rmd file (to diagnose connection db connection errors) and sending those chunks to stdout or stderr is the only way to do that (without logging in to the container which, in my use case (i.e., deployed to AWS) is impossible)

I've reviewed the knitr chunk options and there doesn't seem to be any option I can set to force the chunk output to a file or to stdout or stderr.

Is there some way I can write all of the chunk output to stdout or stderr inside of the render() function? This several-years old question is similar to mine (if not identical) but the accepted answer does not fit my use case

Answer 1

There are two approaches I would take to this problem - depending on what you want.

If you want to see the stderr/stdout output, as you would in a console, the simplest way to accomplish this is to use a shell script to render your doc, and pipe the output to text. - The littler package contains an example shell script render.r that is very useful for this purpose. After installing littler, and ensuring that the scripts are available on your path, you can run:

render.r test.rmd > output.txt

or

render.r test.rmd > output.txt 2>&1

to include stderr in the output file.

However, I've found that this console output is often not sufficiently detailed for debugging and troubleshooting purposes.

So, another option is to edit the Rmd file to log more details about progress/errors & to direct the logger output to an external file.

This is particularly helpful in the context of a cloud or docker-container compute environment since the log can be directed to a datastore allowing for real-time logging, and searching logs across many jobs. Personally, I do this using the futile.logger package.

Using a logger created by futile.logger, this would work as follows:

1. In your Rmd file or in functions called by code in your Rmd file, redirect important error & warning messages to the logger. The best way to do this is the topic of another question, and in my experience varies by task.

   • At a minimum, this results in inserting a series of R commands like follows in your Rmd file:

     library(futile.logger)
     flog.info('Querying data .. ')
     data <- tryCatch(dbGetQuery(...),
                      warning = function(war) {flog.warn(war)}, 
                      error = function(err) {flog.error(err)},
                      ...)
     

     possibly editing the logged messages to provide more context.
   • A more thorough solution will apply globally either to a code chunk or a file. I have not tested this personally in the context of an Rmd, but it might involve using withCallingHandlers, or changing options(error = custom_logging_function).

2. In your R session, before rendering your Rmd, redirect the logger output to a file or to your desired destination.

   • This looks something like:

     library(futile.logger)
     flog.logger(name = 'ROOT', 
                 appender = appender.file('render.log'))
     rmarkdown::render('my_document.Rmd')
     

3. As the document is rendering, you will see the logger output printed to the render.log file.

I would note that, while I actively use the futile.logger package, this package has now been deprecated into a new iteration of it, called logger. I haven't tried this approach specifically with logger, but I suspect it would work just as well if not better. The differences between logger & futile.logger are described very well in this vignette on migration, from the logger docs.

Answer 2

I believe the usual way to do this is to run a separate R instance and capture its output. Without any error checking:

output <- system2("R","-e \"rmarkdown::render('test.Rmd')\"",
                  stdout = TRUE, stderr = TRUE)

This puts all of the output into the output vector. Maybe you can run analysis code in the docker container to look for problems.