CODEWITHSUNDEEP
r
user1969717
user1969717

Reputation: 85

Loop through subfolders and extract data from CSV files

I am trying to loop through all the subfolders of my wd, list their names, open 'data.csv' in each of them and extract the second and last value from that csv file.

The df would look like this :

Name_folder_1   2nd value   Last value
Name_folder_2   2nd value   Last value
Name_folder_3   2nd value   Last value

For now, I managed to list the subfolders and each of the file (thanks to this thread: read multiple text files from multiple folders) but I struggle to implement (what I'm guessing should be) a nested loop to read and extract data from the csv files.

parent.folder <- "C:/Users/Desktop/test"
setwd(parent.folder)

sub.folders1 <- list.dirs(parent.folder, recursive = FALSE)
r.scripts <- file.path(sub.folders1)

files.v <- list()
for (j in seq_along(r.scripts)) {
  files.v[j] <- dir(r.scripts[j],"data$")
}

Any hints would be greatly appreciated !

EDIT :

I'm trying the solution detailed below but there must be something I'm missing as it runs smoothly but does not produce anything. It might be something very silly, I'm new to R and the learning curve is making me dizzy :p

  lapply(files, function(f) {
  dat <- fread(f) # faster
  dat2 <- c(basename(dirname(f)), head(dat$time, 1), tail(dat$time, 1))
  write.csv(dat2, file = "test.csv")
})

Upvotes: 0

Views: 2129

Answers (1)

Martin Schmelzer
Martin Schmelzer

Reputation: 23919

Not easy to reproduce but here is my suggestion:

library(data.table)
files <- list.files("PARENTDIR", full.names = T, recursive = T, pattern = ".*.csv")

lapply(files, function(f) {
  dat <- fread(f) # faster
  # Do whatever, get the subfolder name for example
  basename(dirname(f))
})

You can simply look recursivly for all CSV files in your parent directory and still get their corresponding parent folder.

Upvotes: 1

Related Questions