CODEWITHSUNDEEP
linuxbashshellunix
user2233834
user2233834

Reputation: 373

Memory Issues : Shell Scripting

I have 5 files in a folder a.csv b.csv ....

I need to combine these files into one file called X.csv and in future keep merging the contents of X.csv with a,b,c etc.

Even with only two lines in each file I have error message saying that there is not enough memory - its essentially only copying 10 lines across. Im using the following command : 

 paste -d, *.csv >> X.csv.

However when i use 

 paste -d, *.csv > X.csv

There are no memory issues. This however I canont use since i need the information on the X.CSV file previously as well so I should only append not edit contents of the whole file.

Would anyone know how I achieve this? These are ',' separated CSV files and I would like to not copy the header (Row 1) which names the columns more than once.

I use MAC OSX Mavericks 8 GB Ram.

Thank You :)

Upvotes: 0

Views: 167

Answers (2)

jimm-cl
jimm-cl

Reputation: 5422

As @loreb said, *.csv is matching X.csv as well. You can avoid that using a proper globbing variable, but that depends if your shell is capable of handling that... I know bash can, and you posted your question with the bash tag, but you said you are using Mac OSX - well, I guess you can try it anyway. 

paste -d, [a-z]*.csv >> X.csv

That will run the paste command for the files named from a.csv to z.csv, lowercase, so it will not take X.csv, which I think is the current problem.

Upvotes: 2

l0b0
l0b0

Reputation: 58828

If you have a lot of data, and in the absence of a tool which keeps track of the line number at which processing last stopped in [a-z].csv (which I don't think exists) you can use the following process:

  1. Ensure that the writing program is not going to write to the CSV files anymore. Some possible ways to do this:
    • Move the files to a read-only filesystem.
    • Stop the file writing program.
    • Somehow force the program to start writing to a new file descriptor.
  2. Rename or move the files if necessary to avoid the writer opening them again.
  3. Restart the writer if necessary.
  4. paste -d, /temporary_directory/*.csv >> /final_destination/X.csv

Upvotes: 0

Related Questions