CODEWITHSUNDEEP
r
L.England
L.England

Reputation: 173

Reading multiple RDS files

I have a directory with multiple RDS files (300+) that I would like to read and combine, these RDS files share the same basic format but have different number of rows & a few different columns in each file. I have the simple code to read one RDS file (All files have same "Events-3digitnumber-4digitnumber-6digitnumber.RDS")

    mydata <- readRDS("Events-104-2014-752043.RDS")

Being new to data science I'm sure this simple answer that I'm missing but would I have to use something like list.files() and either lapply or some for loop.

Upvotes: 10

Views: 17827

Answers (4)

TomTom
TomTom

Reputation: 11

complementing FMM's answer above, you may need to include the "full.names=TRUE" in the list.files command to allow map_dfr to read it properly, depending on the path to your files.

df <- list.files(pattern = ".RDS", full.names=T)%>%
 map_dfr(readRDS)

Upvotes: 1

FMM
FMM

Reputation: 2005

Just to add a tidyverse answer:

library(tidyverse)

df <- list.files(pattern = ".RDS") %>%
  map(readRDS) %>% 
  bind_rows()

Update:

It is advised to use map_dfr for binding rows and map_dfc for binding columns, much more efficient:

df <- list.files(pattern = ".RDS") %>%
  map_dfr(readRDS)

Upvotes: 18

jwarz
jwarz

Reputation: 531

Because the solution from FMM did not work for me with huge data sets, I replaced bind_rows() with data.table::rbindlist():

library(tidyverse)
library(data.table)

df <- list.files(pattern = ".rds") %>%
      map(readRDS) %>% 
      data.table::rbindlist()

Upvotes: 5

sbstn
sbstn

Reputation: 628

First a reproducible example:

data(iris)
# make sure that the two data sets (iris, iris2) have different columns
iris2 = copy(iris)
iris2$Species2 = iris2$Species
iris2$Species = NULL

saveRDS(iris, "Events-104-2014-752043.RDS")
saveRDS(iris2, "Events-104-2015-782043.RDS")

Now you need to

  1. find all file names
  2. read the data
  3. combine the data to one table (if you want that)

I would use data.table::rbindlist because it handles differing columns for you when you set fill = TRUE:

require(data.table)
files = list.files(path = '.', pattern = '^Events-[0-9]{3}-[0-9]{4}-[0-9]{6}\\.RDS$')
dat_list = lapply(files, function (x) data.table(readRDS(x)))
dat = rbindlist(dat_list, fill = TRUE)

Upvotes: 4

Related Questions