CODEWITHSUNDEEP
pythongoogle-cloud-firestore
mitrik_bnr
mitrik_bnr

Reputation: 37

Importing data from firestore to python

I need to transfer some collections from firestore to pandas dataframe for analysis ans have some problems

Method from firebase docs :

docs = db.collection(COLLECTION_NAME).stream()

for doc in docs:
    print(f'{doc.id} => {doc.to_dict()}')

my collection contains 100k elements, and when iterator at loop is near 50k i receive an error :

'_UnaryStreamMultiCallable' object has no attribute '_retry'

I load only one collection, wout parallel downloads

Upvotes: 1

Views: 1892

Answers (1)

Divyani Yadav
Divyani Yadav

Reputation: 1142

As mentioned in the link_1 and link_2, you can check for the syntax of importing collections from firestore to pandas dataframe.

Example 1 :

import pandas as pd

ref = db.collection(u'user')
docs = ref.stream()

items = list(map(lambda x: {**x.to_dict(), 'id': x.id}, docs))

df = pd.DataFrame(items) # , columns=['id', 'email']
df.set_index('id', inplace=True)

Example 2:

import pandas as pd
from google.cloud import firestore

db = firestore.Client()
users = list(db.collection(u'users').stream())

users_dict = list(map(lambda x: x.to_dict(), users))
df = pd.DataFrame(users_dict)

For more information, You can refer to the Stackoverflow thread1 and thread2:

Upvotes: 2

Related Questions