CODEWITHSUNDEEP
rmemory-leaksgeospatialr-sp
Dave_L
Dave_L

Reputation: 383

Large SpatialPolygonsDataFrame causing memory leak

I have a program that loads a large SpatialPolygonsDataFrame object (1.4Gb) into memory performs some analysis, then attempts to remove the object. However, a look at the system memory using the system command free shows that the object remains until the R session is reset. I can reproduce the memory leak using the rworldmap and rworlxtra packages to make a large list of world maps, each a SpatialPolygonsDataFrame, then trying to remove them:

install.packages("sp")
install.packages("rworldmap")
install.packages("rworldxtra")
library(sp)
library(rworldmap)
library(rworldxtra)

these.maps.large <- lapply(1:100, function(x) assign(paste0("a_", x), getMap(resolution = "high")))
these.maps.smaller <- lapply(1:20, function(x) assign(paste0("a_", x), getMap(resolution = "high")))
# This frees the memory
rm(list="these.maps.smaller")
gc(reset=T)
# This fails to free the memory
rm(list="these.maps.large")
gc(reset=T)

EDIT Here is the output for calling system2("free") after each stage.

Restarting R session...

> library(sp)
> library(rworldmap)
### Welcome to rworldmap ###
For a short introduction type :      vignette('rworldmap')
> library(rworldxtra)
> system2("free")
              total        used        free      shared  buff/cache   available
Mem:      131987656     1386468   118712292      540008    11888896   129731040
Swap:       4194300     3505464      688836
> these.maps.large <- lapply(1:100, function(x) assign(paste0("a_", x), getMap(resolution = "high")))
> system2("free")
              total        used        free      shared  buff/cache   available
Mem:      131987656     2708040   117390660      540008    11888956   128409404
Swap:       4194300     3505464      688836
> rm(list="these.maps.large")
> gc(reset=T)
         used (Mb) gc trigger  (Mb) max used (Mb)
Ncells 585803 31.3    9601876 512.8   585803 31.3
Vcells 711198  5.5   96623732 737.2   711198  5.5
> system2("free")
              total        used        free      shared  buff/cache   available
Mem:      131987656     2708428   117390424      540008    11888804   128409168
Swap:       4194300     3505464      688836

Restarting R session...

> library(sp)
> library(rworldmap)
### Welcome to rworldmap ###
For a short introduction type :      vignette('rworldmap')
> library(rworldxtra)
> system2("free")
              total        used        free      shared  buff/cache   available
Mem:      131987656     1386696   118711988      540008    11888972   129730744
Swap:       4194300     3505464      688836
> these.maps.smaller <- lapply(1:20, function(x) assign(paste0("a_", x), getMap(resolution = "high")))
> system2("free")
              total        used        free      shared  buff/cache   available
Mem:      131987656     1699628   118399100      540008    11888928   129417836
Swap:       4194300     3505464      688836
> rm(list="these.maps.smaller")
> gc(reset=T)
         used (Mb) gc trigger  (Mb) max used (Mb)
Ncells 702817 37.6    2564361 137.0   702817 37.6
Vcells 966452  7.4   21638748 165.1   966452  7.4
> system2("free")
              total        used        free      shared  buff/cache   available
Mem:      131987656     1699612   118399116      540008    11888928   129417852
Swap:       4194300     3505464      688836

Has anyone got any idea why this is the case and any way it might be possible to remove one of these large sp objects without having to reset the session?

R version 3.2.3 (2015-12-10) Platform: x86_64-redhat-linux-gnu (64-bit) Running under: Scientific Linux 7.2 (Nitrogen)

Upvotes: 1

Views: 96

Answers (1)

Edzer Pebesma
Edzer Pebesma

Reputation: 4121

You don't show the output. I see:

> rm(list="these.maps.smaller")
> gc(reset=T)
            used  (Mb) gc trigger   (Mb)  max used  (Mb)
Ncells   7782803 415.7   14442815  771.4   7782803 415.7
Vcells 113371012 865.0  184235296 1405.7 113371012 865.0
> # This fails to free the memory
> rm(list="these.maps.large")
> gc(reset=T)
         used (Mb) gc trigger   (Mb) max used (Mb)
Ncells 524121   28   11554252  617.1   524121   28
Vcells 649283    5  147388236 1124.5   649283    5

which suggest that removing these large maps releases most memory; the value is nearly equal to that obtained in a fresh session after loading the packages.

Upvotes: 2

Related Questions