CODEWITHSUNDEEP
rfilecsvappend
SriniShine
SriniShine

Reputation: 1139

How to append multiple files in R

I'm trying to read a list of files and append them into a new file with all the records. I do not intend to change anything in the original files. I've tried couple of methods.

Method 1: This methods creates a new file but at each iteration the previous file gets added again. Because I'm binding the data frame recursively.

files <- list.files(pattern = "\\.csv$")

  #temparary data frame to load the contents on the current file
  temp_df <- data.frame(ModelName = character(), Object = character(),stringsAsFactors = F)

  #reading each file within the range and append them to create one file
  for (i in 1:length(files)){
    #read the file
    currentFile = read.csv(files[i])

    #Append the current file
    temp_df = rbind(temp_df, currentFile)    
  }

  #writing the appended file  
  write.csv(temp_df,"Models_appended.csv",row.names = F,quote = F)

Method 2: I got this method from Rbloggers . This methods won't write to a new file but keeps on modifying the original file. 

multmerge = function(){
  filenames= list.files(pattern = "\\.csv$")
  datalist = lapply(filenames, function(x){read.csv(file=x,header=T)})
  Reduce(function(x,y) {merge(x,y)}, temp_df)

}

Can someone advice me on how to achieve my goal?

Upvotes: 8

Views: 26646

Answers (7)

Herv&#233; Pag&#232;s
Herv&#233; Pag&#232;s

Reputation: 56

The concatenate_files() function below will concatenate a set of files to a single file. The kind of input files (e.g. csv, fasta, etc...) doesn't matter. Also the function doesn't load an entire input file in memory before appending it to the output. Instead small chunks of data are loaded in memory, one chunk at a time. The chunk size is controlled by the n argument (50k bytes by default).

concatenate_files <- function(files, out=stdout(), n=50000L)
{
    stopifnot(is.character(files))
    if (is.character(out)) {
        out <- file(out, "wb")
        on.exit(close(out))
    }
    for (f in files) {
        con <- file(f, "rb")
        while (TRUE) {
            bytes <- readBin(con, what=raw(), n=n)
            if (length(bytes) == 0L)
                break
            writeBin(bytes, out)
        }
        close(con)
    }
}

Upvotes: 0

Bradford
Bradford

Reputation: 107

If using Windows, it's easy to do this using the command prompt.

Start -> Run -> type "cmd" and hit return key

cd <path to folder>
copy /b *.txt <outputname>.txt

Concrete example:

cd C:\User\danny\docs\folder_with_txt files
copy /b *.txt concatenated.txt

Note that if you're changing drive letters to do this before cd

D:\> c:
C:\> cd C:\User\danny\docs\folder_with_txt files
copy /b *.txt concatenated.txt

Upvotes: 0

yuan-ning
yuan-ning

Reputation: 607

There's an easy answer with rbind_list() now!

For instance:

dataFiles = map(Sys.glob("*.csv"), read.csv)

or however you're reading the files into a list

dat = rbind_list(dataFiles)

and dat will be what you're looking for!

Upvotes: 2

Ferroao
Ferroao

Reputation: 3043

Try this for your ListOfFileNames:

ListOfFileNames<-list.files(pattern=".txt")
outFile <- file("all.txt", "w")
for (i in ListOfFileNames){
    x <- readLines(i)
    writeLines(x, outFile) # in the link the 1st and last line are skipped
}
close(outFile)

Source: https://r.789695.n4.nabble.com/R-Read-multiple-text-files-and-combine-into-single-file-td817344.html

Upvotes: 2

KeelyD
KeelyD

Reputation: 161

Or you could go ahead and use shell commands in R:

system2("cat", args = "*.csv", stdout = "appendedfiles.csv")

This would be for unix based systems; I'm not sure what you would do for windows.

Upvotes: 5

jogo
jogo

Reputation: 12559

it could look like this:

files <- list.files(pattern = "\\.csv$")

DF <-  read.csv(files[1])

#reading each file within the range and append them to create one file
for (f in files[-1]){
  df <- read.csv(f)      # read the file
  DF <- rbind(DF, df)    # append the current file
}
#writing the appended file  
write.csv(DF, "Models_appended.csv", row.names=FALSE, quote=FALSE)

or short:

files <- list.files(pattern = "\\.csv$")

DF <-  read.csv(files[1])
for (f in files[-1]) DF <- rbind(DF, read.csv(f))   
write.csv(DF, "Models_appended.csv", row.names=FALSE, quote=FALSE)

Upvotes: 12

m0nhawk
m0nhawk

Reputation: 24188

You can use this to load everything into one data set.

dataset <- do.call("rbind", lapply(file.list, FUN = function(file) {
  read.table(file, header=TRUE, sep="\t")
}))

And then just save with write.csv.

Upvotes: 7

Related Questions