How to loop twice over files' names in R

Question

Let's say I am running a study on two subjects (in reality, 20 of them). Since each subject generates 27 files that I need to combine to generate additional 9 files, I would like to automate this process!

I have one factor varying on nine levels: AA, AB, AM, BA, BB, BM, MA, MB, MM.

For each treatment I get three output files, for example for the AA treatment I get: AA1.csv, AA1.txt and AA1log.txt.

I will need to run a script (let's call it R1) on these files; it will combine them together in a summary file. Then I will need to run another script (I will call it R2) on all the summary files I have generated.

All the output files for all the subjects are in one folder, "data".

(for the R example, thank you to @ManuelBickel)

# make sure you are in a safe directory!

### Generate the toy data ###

# I define the main directories I need
dir_project = "test"
dirs = list(
  dir_project = dir_project
  ,dir_data = paste0(dir_project, "/data")
  ,dir_summary = paste0(dir_project, "/summary")
  ,dir_plots= paste0(dir_project, "/plots")
)
# create dirs
lapply(dirs, dir.create)

# create some exemplary data and write it in dir
m = matrix(1:4, nrow = 2)
data = list(AA = m, AB = m, AM = m
            ,BA = m, BB = m, BM = m,
            MA = m, MB = m, MM =m)

# generate the csv files for subject 1 and 2
for (i in 1:length(data)) {
  write.csv(data[[i]], file = paste0(dirs[["dir_data"]], "/", names(data[i]), "1.csv"))  
}

for (i in 1:length(data)) {
write.csv(data[[i]], file = paste0(dirs[["dir_data"]], "/", names(data[i]), "2.csv"))  
}

# Generate the .txt files for subjects 1 and 2
for (i in
1:length(data)) {
  write.table(data[[i]], file = paste0(dirs[["dir_data"]], "/", names(data[i]), "1.txt"))  
}

for (i in 1:length(data)) {
  write.table(data[[i]], file = paste0(dirs[["dir_data"]], "/", names(data[i]), "2.txt"))  
}

 # Generate the log.txt files for subjects 1 and 2
for (i in 1:length(data)) {
  write.table(data[[i]], file = paste0(dirs[["dir_data"]], "/", names(data[i]), "1log.txt"))  
}

for (i in 1:length(data)) {
  write.table(data[[i]], file = paste0(dirs[["dir_data"]], "/", names(data[i]), "2log.txt"))  
}

So the following are the files I have in my data folder:

list.files(dirs[["dir_data"]])

# [1] "AA1.csv"    "AA1.txt"    "AA1log.txt" "AA2.csv"    "AA2.txt"    "AA2log.txt" "AB1.csv"    "AB1.txt"    "AB1log.txt"
# [10] "AB2.csv"    "AB2.txt"    "AB2log.txt" "AM1.csv"    "AM1.txt"    "AM1log.txt" "AM2.csv"    "AM2.txt"    "AM2log.txt"
# [19] "BA1.csv"    "BA1.txt"    "BA1log.txt" "BA2.csv"    "BA2.txt"    "BA2log.txt" "BB1.csv"    "BB1.txt"    "BB1log.txt"
# [28] "BB2.csv"    "BB2.txt"    "BB2log.txt" "BM1.csv"    "BM1.txt"    "BM1log.txt" "BM2.csv"    "BM2.txt"    "BM2log.txt"
# [37] "MA1.csv"    "MA1.txt"    "MA1log.txt" "MA2.csv"    "MA2.txt"    "MA2log.txt" "MB1.csv"    "MB1.txt"    "MB1log.txt"
# [46] "MB2.csv"    "MB2.txt"    "MB2log.txt" "MM1.csv"    "MM1.txt"    "MM1log.txt" "MM2.csv"    "MM2.txt"    "MM2log.txt"

Now I need my code to pick the files: AA1.csv, AA1.txt and AA1log.txt and run the script R1 on them.

The script R1 will generate as output one csv file that will go in the folder "data" as "summaryAA1_csv". It will also generate 32 png. files (AA1_1.png, AA1_2.png and so on) that will go into a subfolder "AA1" in the folder "plots".

Then I will pick all the summary files for subject 1 out of the folder "data" and run the script R2.

Besically first I need to pick all the datasets produced by subject 1; then I need to pick the ones generated by the same treatment (all the AAs first, then the ABs etc.). Once I have gone trough the nine treatments, I move to subject 2.

Let's say this is what R1 is doing:

temp = read.csv("test/data/AA1.csv", sep=",", row.names=1)
temp1 <- as.matrix(temp) 
temp2 <- read.table("test/data/AA1.txt")
temp3 <- read.table("test/data/AA1log.txt")
summaryAA1 <- temp1 + temp2 + temp3
summaryAA1

As I wrote my R1 code also generates plots (32 for each treatment!) that go in a different folder

dir.create("test/plots/AA1plots")
png(filename="test/plots//AA1plots/AA1_1_plot.png")
plot(summaryAA1)
dev.off()

My question is how I make my code select the files twice; first select the files that refer to the same treatment (AA) and the same subject number; once all the treatments have been run, move to the files that refer to the same treatment for the second subject.

I am also open to suggestions about a more convenient naming system that may make the looping more convenient.

Answer 1

Consider organizing your inputs (list of subject and treatment combinations) and processes (R1 and R2). Then call them appropriately:

subjects <- c(1, 2)
treatments <- c("AA", "AB", "AM", "BA", "BB", "BM", "MA", "MB", "MM")

r1_list <- as.vector(sapply(subjects, function(x,y) paste0(y,x), treatments))
# [1] "AA1" "AB1" "AM1" "BA1" "BB1" "BM1" "MA1" "MB1" "MM1" "AA2" "AB2" "AM2" "BA2" "BB2" "BM2" "MA2" "MB2" "MM2"

r2_list <- sapply(subjects, function(x,y) paste0(y,x), treatments, simplify = FALSE)
r2_list
# [[1]]
# [1] "AA1" "AB1" "AM1" "BA1" "BB1" "BM1" "MA1" "MB1" "MM1"

# [[2]]
# [1] "AA2" "AB2" "AM2" "BA2" "BB2" "BM2" "MA2" "MB2" "MM2"

R1 Script

setwd("test")

my_func1 <- function(f){
    temp = read.csv(paste0("data/", f, ".csv"), row.names=1)
    temp1 <- as.matrix(temp) 
    temp2 <- read.table(paste0("data/", f, ".txt"))
    temp3 <- read.table(paste0("data/", f, "log.txt"))

    # SUMMARIES
    summary_all <- temp1 + temp2 + temp3
    summary_data <- read.csv(paste0("summary", f, ".csv"))

    ...

    # IMAGES
    for (i in seq(1,32)) {
        dir.create(paste0("plots/", f, "plots"))
        png(filename=paste0("plots/", f, "plots/", f, "_", i, "_plot.png"))
        plot(...)
        dev.off()
    }
}

# CREATE ALL SUMMARY AND IMAGE FILES
for (j in r1_list) my_func1(j)

R2 Script

my_func2 <- function(items){
    files <- paste0("summary", items, ".csv")

    # READ ALL SUMMARY FILES INTO A LIST OF DATA FRAMES
    df_list <- lapply(files, read.csv)

    # PROCESS LIST
    ...    
}

# PROCESS SUMMARY FILES
for (j in r2_list) my_func2(j)