CODEWITHSUNDEEP
r
lpchaparro
lpchaparro

Reputation: 129

ffdfappend from ffbase

R friends.

Due to the size of my data i´m using the ff and ffbase packages. My question is about the ffdfappend(). I have different ffsf files saved as .csv. What I need to do is load all the files and apply the ffdfd function in order to create just one big ffdf file. I do this as a loop:

files<-list.files()  
for (i in 1:length(files)){
fdata = read.csv.ffdf(file=files[i], first.rows=400, colClasses=NA)   
colnames(fdata)<-c('fecha','juliano','UTM_X','UTM_Y','temp','pp','f_ocur', 'altitud')  
if (i == 1){data=fdata} else {data<-ffdfappend(data, fdata, adjustvmode=F)}  
delete(fdata); rm(fdata)
}

I can read the files in the folder easily. The problem is when I want to “append” them in one ffdf object. R tells me this:

Error en sort.list(y) : 'x' must be atomic for 'sort.list'  
Have you called 'sort' on a list?***

I don’t understand what is telling me, so I´m unable to solve the problem. 

str(files)
chr [1:6] "102c1b481014.csv" "102c2211788.csv" "102c32963052.csv" "102c32f1798.csv" "102c3a2517f1.csv" "102c4e7513f0.csv"

If anyone can help me understand or solve this problem I´d appreciate it

Upvotes: 1

Views: 595

Answers (1)

user1600826
user1600826

Reputation:

ffdfappend is basically for appending a data.frame to an ffdf. I don't know if this is an option, but loading each data.frame in RAM before appending to the ffdf will work, as is done below.

files<-list.files()  
for (i in 1:length(files)){
  fdata = read.csv.ffdf(file=files[i], first.rows=400, colClasses=NA)   
  colnames(fdata)<-c('fecha','juliano','UTM_X','UTM_Y','temp','pp','f_ocur', 'altitud')  
  if (i == 1){data=fdata} else {data<-ffdfappend(data, fdata[,], adjustvmode=TRUE)}  
  delete(fdata); rm(fdata)
}

Another option is to use the x argument of read.csv.ffdf. As shown below. But your data really needs to have the same structure.

files<-list.files()  
fdata = read.csv.ffdf(file=files[1], first.rows=400, colClasses=NA)
for (i in 2:length(files)){
  fdata = read.csv.ffdf(x = fdata, file=files[i], first.rows=400, colClasses=NA)   
}

Upvotes: 3

Related Questions