CODEWITHSUNDEEP
rmemoryspatialrasterr-raster
rachell
rachell

Reputation: 19

"Failure during raster IO" possibly related to raster size?

I am with very large rasters in R. I have a code that I have been able to run before, with the same large rasters, but when I try to run my code again for reproducibility, I am getting the same error "Failure during raster IO".

For instance, in this section of a code I am trying to break a large raster (~ 5 GB) into smaller tiles so that I can speed up processing time in R.

large_raster <- stack("D:/large_raster.tif")
tileno <- 2 # split the large raster into 2 tiles

cl <- makeCluster(3)
registerDoSNOW(cl)
foreach(i = 1:nlayers(large_raster))%dopar%{
  library(raster)
  library(SpaDES)
  removeTmpFiles(h=0)
  newraster <- splitRaster(large_raster[[i]],nx=tileno,ny=1)
  for(j in 1:length(newraster)){
    if(i<10){
      writeRaster(newraster[[j]], filename=paste("D:/large_raster_layer0", i, "_tile", j, ".tif",sep=""))
    }else{
      writeRaster(newraster[[j]],filename=paste("D:/large_raster_layer", i, "_tile", j, ".tif",sep = ""))
    }
  }  
}
stopCluster(cl)

#stack tiles
dir <- ("D:/")
for(j in 1:tileno){
  tiles<-list.files(dir,pattern=paste("tile",j,sep=""))
  for(i in 1:length(tiles)){
    r<-raster(paste(dir,tiles[i],sep=""))
    if(i==1){s<-stack(r)
    }else{s<-stack(s,r)}
  }
  writeRaster(s, filename = paste("D:/large_raster_tile", j, ".tif", sep=""))
}

But I get the error

Error in rgdal::getRasterData(object@file@con, offset = offs, region.dim = reg,  : 
  Failure during raster IO

If I skip the tiling step and just end up working with my 5 GB large_raster I end up getting the error again in this code, where I am creating a new raster called wfps_growSeas by overlaying 2 other rasters called dswe2wfps and swt:

cl <- makeCluster(3)
registerDoSNOW(cl)

wfps_growSeas <- foreach(j = 1:nlayers(dswe2wfps))%dopar%{
  library(raster)
  overlay(dswe2wfps[[j]],swt[[j]], fun=function (x,y){
    x[y[]<=5]<-0
    return (x)},filename=paste("D:/temp2delete/raster",j, "_", info, ".tif",sep=""),overwrite=TRUE)
}  
stopCluster(cl)

wfps_growSeas <- stack(wfps_growSeas)

And I get the error:

Error in { : task 1 failed - "Failure during raster IO

Strangely, these errors has happened recently. A few weeks ago, I was able to run this exact code with these exact same large rasters, and currently, I am able to run these coded using smaller rasters. I am using R-3.6.2, but colleagues have suggested this error is usually related to sheer raster size and R's memory. This post suggests this error is due to corrupt rasters, but I don't think that's the problem. Any suggestions?

Upvotes: 0

Views: 3118

Answers (2)

rachell
rachell

Reputation: 19

The issue was solved by changing my folders (and the rasters saved in the folders) from a D drive (external 2 TB solid state hard drive) to my computer's C drive. I can still run codes working with smaller raster sizes directly on my D drive (which, in this case, actually still had plenty of memory left on it).

Upvotes: 0

Robert Hijmans
Robert Hijmans

Reputation: 47081

probably because you are out of disk space, mostly like in the temp folder. Clean the temp folder and/or set it to a different location. See ?rasterOptions

Upvotes: 1

Related Questions