CODEWITHSUNDEEP
rpdfggplot2heatmap
user213544
user213544

Reputation: 2126

Speed up rendering of large heatmap from ggplot in R

I am trying to plot a large heatmap, generated with ggplot, in R. Ultimately, I would like to 'polish' this heat map using Illustrator.

Sample code:

# Load packages (tidyverse)
library(tidyverse)

# Create dataframe
df <- expand.grid(x = seq(1,100000), y = seq(1,100000))

# add variable: performance
set.seed(123)
df$z <- rnorm(nrow(df))

ggplot(data = df, aes(x = x, y = y)) +
  geom_raster(aes(fill = z))

Although I save the plot as a vectorized image (.pdf; that is not that large), the pdf is loading very slowly when opening. I expect that every individual point in the data frame is rendered when opening the file.

I have read other posts (e.g. Data exploration in R: display heatmap of large matrix, quickly?) that use image() to visualize matrices, however I would like to use ggplot to modify the image.

Question: How do I speed up the rendering of this plot? Is there a way (besides lowering the resolution of the plot), while keeping the image vectorized, to speed this process up? Is it possible to downsample a vectorized ggplot?

Upvotes: 0

Views: 3354

Answers (2)

Jon Spring
Jon Spring

Reputation: 66415

The first thing I tried was stat_summary_2d to get average binning, but it seemed slow and also created some artifacts on the right and top edges:

library(tidyverse)
df <- expand.grid(x = seq(1,1000), y = seq(1,1000))
set.seed(123)
df$z <- rnorm(nrow(df))
print(object.size(df), units = "Mb")
#15.4 Mb

ggplot(data = df, aes(x = x, y = y, z = z)) +
  stat_summary_2d(bins = c(100,100)) +  #10x downsample, in this case
  scale_x_continuous(breaks = 100*0:10) +
  labs(title = "stat_summary_2d, 1000x1000 downsampled to 100x100")

Even though this is much smaller than your suggested data, this still took about 3 seconds to plot on my machine, and had artifacts on the top and right edges, I presume due to those bins being smaller ones from the edges, leaving more variation.

enter image description here

It got slower from there when I tried a larger grid like you are requesting.

(As an aside, it may be worth clarifying that a vector graphic file like a PDF, unlike a raster graphic, can be resized without loss of resolution. However, in this use case, the output is 10,000 megapixel raster file, far beyond the limits of human perception, that is getting exported into a vector format, where each "pixel" becomes a very tiny rectangle in the PDF. That use of a vector format could be useful for certain unusual cases, like if you ever need to blow up your heatmap without loss of resolution onto a gigantic surface, like a football field. But it sounds like in this case it might be the wrong tool for the job, since you're putting heaps of data into the vector file that won't be perceptible.)

What worked more efficiently was to do the averaging with dplyr before ggplot. With that, I could take a 10k x 10k array and downsample it 100x before sending to ggplot. This necessarily reduces the resolution, but I don't understand the value in this use case of preserving resolution beyond human abilities to perceive it.

Here's some code to do the bucketing ourselves and then plot the downsampled version:

# Using 10k x 10k array, 1527.1 Mb when initialized
downsample <- 100
df2 <- df %>%
  group_by(x = downsample * round(x / downsample),
           y = downsample * round(y / downsample)) %>%
  summarise(z = mean(z))

ggplot(df2, aes(x = x, y = y)) +
  geom_raster(aes(fill = z)) +
  scale_x_continuous(breaks = 1000*0:10) +
  labs(title = "10,000x10,000 downsampled to 100x100")

enter image description here

Upvotes: 6

moodymudskipper
moodymudskipper

Reputation: 47300

Your reproducible example just shows noise so it's hard to know what kind of output you would like.

One way would be to follow @dww's suggestion and use geom_hex to show aggregated data.

Another way, as you ask "Is it possible to downsample a vectorized ggplot?", is to use dplyr::sample_frac or dplyr::sample_n in the data argument of your geom_raster. I have to take a smaller sample than in your example though or I can't build the df.

library(tidyverse)

# Create dataframe
df <- expand.grid(x = seq(1,1000), y = seq(1,1000))

# add variable: performance
set.seed(123)
df$z <- rnorm(nrow(df))

ggplot(data = df, aes(x = x, y = y)) +
  geom_raster(aes(fill = z), . %>% sample_frac(0.1))

If you want to start from your high resolution ggplot object you can do for the same effect:

gg <- ggplot(data = df, aes(x = x, y = y)) +
  geom_raster(aes(fill = z)) 

gg$data <-  sample_frac(gg$data,0.1)
gg

Upvotes: 3

Related Questions