CODEWITHSUNDEEP
rdataframedata.table
KTY
KTY

Reputation: 719

Comparing speed of fread vs. read.table for reading the first 1M rows out of 100M

I have a 14GB data.txt file. I was comparing the speed of fread and read.table by reading the first 1M rows. It looks like fread is much slower although it is not supposed to be. It takes some time until the percentage counts show up.

What could be the reason? I thought it was supposed to be super fast... I am using a Windows OS computer.

Upvotes: 6

Views: 2582

Answers (1)

mnel
mnel

Reputation: 115392

fread mmaps the file. This takes some time, and will map the whole file. This means subsequent "read-ins" will be faster.

read.table does not mmap the whole file. It can read in the file line by line [and stop at line 1000000].

You can see some background on mmap at mmap() vs. reading blocks

The examples in the help from fread highlight this behaiviour

Upvotes: 9

Related Questions