CODEWITHSUNDEEP
firebase
mihirg
mihirg

Reputation: 951

Loading Bulk data in Firebase

I am trying to use the set api to set an object in firebase. The object is fairly large, the serialized json is 2.6 mb in size. The root node has around 90 chidren, and in all there are around 10000 nodes in the json tree.

The set api seems to hang and does not call the callback. It also seems to cause problems with the firebase instance.

Any ideas on how to work around this?

Upvotes: 0

Views: 4805

Answers (1)

Kato
Kato

Reputation: 40582

Since this is a commonly requested feature, I'll go ahead and merge Robert and Puf's comments into an answer for others.

There are some tools available to help with big data imports, like firebase-streaming-import. What they do internally can also be engineered fairly easily for the do-it-yourselfer:

1) Get a list of keys without downloading all the data, using a GET request and shallow=true. Possibly do this recursively depending on the data structure and dynamics of the app.

2) In some sort of throttled fashion, upload the "chunks" to Firebase using PUT requests or the API's set() method.

The critical components to keep in mind here is that the number of bytes in a request and the frequency of requests will have an impact on performance for others viewing the application, and also count against your bandwidth.

A good rule of thumb is that you don't want to do more than ~100 writes per second during your import, preferably lower than 20 to maximize your realtime speeds for other users, and that you should keep the data chunks in low MBs--certainly not GBs per chunk. Keep in mind that all of this has to go over the internets.

Upvotes: 2

Related Questions