CODEWITHSUNDEEP
rdata.tableff
dleal
dleal

Reputation: 2314

Efficient way to read file larger than memory in R

This reference https://www.r-bloggers.com/efficiency-of-importing-large-csv-files-in-r/ compares reading a file using fread versus ffdf. I am currently trying to read a csv file that is abour 60GB while my memory available on RAM is 16GB. It takes about 2 hours to do so. Would you recommend a faster way?

Upvotes: 0

Views: 2390

Answers (3)

Zaki
Zaki

Reputation: 141

disk.frame can chunk the file into smaller parts, uses fst format and you can use data.table or dplyr syntax.

install.packages("disk.frame")

see here for more info: https://diskframe.com/#:~:text=frame%7D%20is%20an%20R%20package,to%20be%20processed%20by%20R.

Upvotes: 0

Dongdong Kong
Dongdong Kong

Reputation: 416

bigmemory works in this situation.

library(bigmemory)
library(biganalytics)
x <- read.big.matrix("airline.csv", type="integer", header=TRUE,
                     backingfile="airline.bin",
                     descriptorfile="airline.desc",
                     extraCols="Age")

Upvotes: 1

Salvador Santos
Salvador Santos

Reputation: 11

Use Data.Table Package.

Seems to be faster Function Fread.

Function: 

"File Name"<- fread("Uploadfile.txt", header =  TRUE, colClasses = "character")

Upvotes: 0

Related Questions