CODEWITHSUNDEEP
rcsvfeather
epo3
epo3

Reputation: 3121

Error: Invalid: File is too small to be a well-formed file - error when using feather in R

I'm trying to use feather (v. 0.0.1) in R to read a fairly large (3.5 GB) csv file with 21178665 rows and 16 columns.

I use the following lines to load the file:

library(feather)
path <- "pp-complete.csv"
df <- read_feather(path)

But I get the following error:

Error: Invalid: File is too small to be a well-formed file

There's no explanation in the documentation of read_feather so I'm not sure what's the problem. I guess this function expects a different file form but I'm not sure what that would be.

Btw, I can read the file with read_csv in readr library but it takes a while.

Upvotes: 0

Views: 1709

Answers (1)

MrFlick
MrFlick

Reputation: 206401

The feather file format is distinct from a CSV file format. They are not interchangeable. The read_feather function cannot read simple CSV files.

If you want to read CSV files quickly, your best bets are probably readr::read_csv or data.table::fread. For large files, it will still usually take a while just to read it from disc.

After you've loaded the data into R, you can create a file in the feather format with write_feather so you can read it with read_feather the next time.

Upvotes: 3

Related Questions