CODEWITHSUNDEEP
meteorfile-uploadamazon-s3mergemeteor-slingshot
Enmanuel Duran
Enmanuel Duran

Reputation: 5118

Processing and Merging files before uploading them to Amazon S3

I'm developing a web platform using meteorjs that will allows me to upload multiple text/plain such as .txt and .csv files to my amazon S3 bucket.

Currently my platform supports multiple files uploads, uploading them sequentially by using a for loop and meteor-slingshot to upload to my s3 directly.

What I'm trying to do:
I need to merge the files into one final file before the uploading to S3 happens so I can upload just one file instead of uploading multiple files (Files must be merged in one).

My ideas:

  1. I've thought about uploading the files into my own server before they're uploaded to S3 and then use something like cat file1.txt file2.txt > merged_file.txt in linux to merge them and retrieve this file somehow, but it'll take longer because we need to upload the files to my server and then upload the merged file to the S3 bucket.

  2. Use the fs.appendFile from node to merge the files, but I don't know if it could be viable because files can be up to 20/25mb and it'll will take some time to read them and append the information.

I think javascript is not probably viable to handle this directly, that's why I'm thinking in sh commands or c++ because of the nodejs core.

Can you recommend me a better way of doing this?
is there a better way to handle this process?

Upvotes: 0

Views: 414

Answers (1)

Joshua Smith
Joshua Smith

Reputation: 6621

You can do this easily in shell.

for i in file1 file2 file3 do cat $i >> combined_file done

I would also say that you might want to consider uploading them as small files and then spin up an aws instance pull them down, merge them, and re-upload them. that way you don't lose the originals and it you can automate this whole process.

Upvotes: 0

Related Questions