How to read and rbind large CSV file efficiently?

Question

I have 20 large CSV (100-150MB each) files i would like to load in R and rbind them in a large file and perform my analysis. Reading each CSV file is performed on one core only and takes about 7 min. I am on 64bit 8-core linux with 16gb RAM so resources should not be an issue.

Is there any way to perform this process more efficiently? I am also open to other (open source linux) software (for example binding the CSV files in a different programm and loading in R) or anything that could make this process faster.

Thank you very much

Answer 1

Maybe you want a function like paste. It's a bash function that merge lines of files.