CODEWITHSUNDEEP
rcsvfor-loop
Bill TP
Bill TP

Reputation: 1097

How to process multiple csv files in R

I have many csv files in three separate folders as follows:

folder1
a1_0023.csv
a2_0034.csv
a3_6163.csv
...
(100 files)

folder2
b1_0023.csv
b2_0034.csv
b3_6163.csv
...
(100 files)

folder3
c1_0023.csv
c2_0034.csv
c3_6163.csv
...
(100 files)

And I have a text file that lists the last four digits:

theLastFourDigits.txt
0023
0034
6163
...
(100 lines)

For the 0023 files, I do a simple job in R:

a <- read.table("D:/folder1/a1_0023.csv", header=FALSE, sep=",")
a <- as.matrix(a)
b <- read.table("D:/folder2/b1_0023.csv", header=FALSE, sep=",")
b <- as.matrix(b)
c <- read.table("D:/folder3/c1_0023.csv", header=FALSE, sep=",")
c <- as.matrix(c)

# Initiate the column vector that contains the results
myanswer <- matrix(0, nrow=100, ncol=1)

# Do a simple job, and store the result in myanswer column
myanswer[1] = nrow(a)*nrow(b)/nrow(c)

I have two questions here: (1) How can we iterate this process for the whole 100 digits? (2) How can we do the multiple jobs if I don't have the theLastFourDigits.txt list file?

EDIT:

I tried something like the following:

setwd("D:/folder1/")
filelist1 <- Sys.glob("*.csv")
setwd("D:/folder2/")
filelist2 <- Sys.glob("*.csv")
setwd("D:/folder3/")
filelist3 <- Sys.glob("*.csv")

for (i in 1:100) {

 setwd("D:/folder1/")
 a <- read.csv(filelist1[i], header=FALSE, sep=",")
 a <- as.matrix(a)
 setwd("D:/folder2/")
 b <- read.csv(filelist2[i], header=FALSE, sep=",")
 b <- as.matrix(b)
 setwd("D:/folder3/")
 c <- read.csv(filelist3[i], header=FALSE, sep=",")
 c <- as.matrix(c)

 nrow(a)*nrow(b)/nrow(c)

}

And the error message is like:

 Error in read.table(file = file, header = header, sep = sep, quote = quote,  : 
  no lines available in input 
3 stop("no lines available in input") 
2 read.table(file = file, header = header, sep = sep, quote = quote, 
    dec = dec, fill = fill, comment.char = comment.char, ...) 
1 read.csv(filelist1[i], header = FALSE, sep = ",")

What am I missing here?

Upvotes: 0

Views: 393

Answers (1)

Jean V. Adams
Jean V. Adams

Reputation: 4784

For question (2), you might find this function useful. I have used it in the past to read in all the csv files in a given folder (Windows 7). You would need to modify the read.csv() arguments as needed for your application. Once all the data from a folder has been read in, you can convert all the data frames to matrices with lapply().

list.csv <- function(mydir, add.source=TRUE) {
    # combine all csv files in a given directory into a single list
    filenames <- list.files(mydir)[grep(".csv$", list.files(mydir))]
    nfiles <- length(filenames)
    # create an empty list where all the files will be stored
    files.list <- vector(mode="list", length=nfiles)
    for(i in 1:nfiles) {
        # read the data into a temporary file
        temp <- read.csv(paste(mydir, filenames[i], sep=""), as.is=TRUE)
        # add a new column identifying the source file
        if(add.source) temp$source <- filenames[i]
        # put the data into the list
        files.list[[i]] <- temp
        }
    files.list
    }

mylist <- list.csv("C:/temp/")

# look at headers from all the data frames
lapply(mylist, head)

# convert all the data frames to matrices
mylistm <- lapply(mylist, as.matrix)

Upvotes: 2

Related Questions