CODEWITHSUNDEEP
jsonnode.jsmongodbcsvmeteor
stackyname
stackyname

Reputation: 526

Storing GBs of JSON-like data in MongoDB

I'm using MongoDB because Meteor doesn't support-officially-anything else.

The main goal is to upload CSV files, parse them in Meteor and import the data to database.

The inserted data size can be 50-60GB or maybe more per file but I can't even insert anything bigger than 16MB due to document size limit. Also, even 1/10 of the insertion takes a lot of time.

I'm using CollectionFS for CSV file upload on the client. Therefore, I tried using CollectionFS for the data itself as well but it gives me an "unsupported data" error.

What can I do about this?

Edit: Since my question creates a confusion about storing data techniques, I want to clear something: I'm not interested in uploading the CSV file; I'm interested in storing the data in the file. I want to collect all user's data in one place and I want to fetch data with the lowest resources.

Upvotes: 2

Views: 205

Answers (2)

perusopersonale
perusopersonale

Reputation: 896

In CollectionFS you can save file directly filesystem, just add the proper package and create your collection like these:

Csv = new FS.Collection("csv", {
    stores: [
        new FS.Store.FileSystem("csv","/home/username/csv")
    ],
    filter: {
        allow: {
            extensions: ['csv']
        }
    }
});

Upvotes: 1

med116
med116

Reputation: 1606

You could insert the csv file as a collection (the filename can become the collection name), with each row of the csv as a document. This will get around the 16 MB per document size limit. You may end up with a lot of collections, but that is okay. Another collection could keep track of the filename to collection name mapping.

Upvotes: 1

Related Questions