CODEWITHSUNDEEP
rdoparallelff
ahmathelte
ahmathelte

Reputation: 623

How to pass ff objects to cluster/nodes

I have my data as ff arrays and stored perfectly on the disk, I'm trying to use parallel computation. Array's names are stored in the data frame DATA.wind in column Smoothed_name

DATA.wind<-subset(DATA,DATA$variable=="u"|DATA$variable=="v")
# Calculate the number of cores
no_cores <- detectCores() - 1
cl <- makeCluster(no_cores)
#Exporting functions and environment to cluster
clusterExport(cl, list("five_points_smoother","DATA.wind"),envir=globalenv())
 
parLapply(cl,
      1:length(DATA.wind$No),function(x){
        medium1<-five_points_smoother(get(DATA.wind$var_name[x]))
        assign(DATA.wind$Smoothed_name[x],ff(medium1,dim =dim(medium1))) 
      })

when i run parLapply i get this error:

`  Error in checkForRemoteErrors(val) : 
   one node produced an error: object 'u_1088632800' not found `

knowing that 'u_1088632800'is one of the arrays name stored in the df.

Upvotes: 0

Views: 68

Answers (0)

Related Questions