CODEWITHSUNDEEP
r
xiaodai
xiaodai

Reputation: 16014

read.csv.ffdf gets stuck in R using the ff and ffbase package

I ran read.csv.ffdf from the ff or ffbase package. But it got stuck with the following message

read.table.ffdf 1..1000 (1000)  csv-read=0.15sec ffdf-write=1.24sec
read.table.ffdf 1001..63015 (62015)  csv-read=9.89sec ffdf-write=8.84sec
read.table.ffdf 63016..125030 (62015)  csv-read=8.18sec ffdf-write=6.74sec
read.table.ffdf 125031..187045 (62015)  csv-read=8.15sec ffdf-write=6.72sec
read.table.ffdf 187046..249060 (62015)  csv-read=8.19sec ffdf-write=6.72sec
read.table.ffdf 249061..311075 (62015)  csv-read=8.05sec ffdf-write=7.13sec
read.table.ffdf 311076..373090 (62015)  csv-read=8.26sec ffdf-write=6.73sec
read.table.ffdf 373091..435105 (62015)  csv-read=8.26sec ffdf-write=6.8sec

The files contains approximately 800000 rows but why does it stop here and I can't run anything from this point onwards?

Upvotes: 0

Views: 924

Answers (1)

user1600826
user1600826

Reputation:

Are you using the transFUN argument of read.csv.ffdf? If yes, make sure the number of rows which you return from transFUN is the same as the number of rows of the data.frame entering transFUN.

Explanation: Importing of read.csv.ffdf stops when a chunk returns less records then the next.rows argument, so if you are subsetting in transFUN, it will stop. transFUN is for adding derived fields or cleaning up data, not subsetting.

Upvotes: 2

Related Questions