CODEWITHSUNDEEP
firebasegoogle-cloud-firestore
miz-k
miz-k

Reputation: 995

Export json from Firestore

As we can download json file at Firebase RTDB console, are there any way to export json file of Firestore collection/document data?

One of my main objectives is to compare data before/after updating document.

Upvotes: 83

Views: 93570

Answers (17)

sun
sun

Reputation: 1068

I wanted a quick and simple export of each collection into separate JSON files, pretty printed, in a human-readable format and including subcollections, for easy data analysis and debugging.

All other proposed solutions seemingly focus on retaining the data structures as they are (for potential reimport and the like), but that is not helpful for actually studying documents and detecting anomalies.

So I created a Python script:

https://github.com/sun/google-firebase-firestore-json-export

$ pip3 install firebase-admin
$ python3 main.py
☑️  Exported to houses.json.
☑️  Exported to parents.json.
☑️  Exported to children.json.
☑️  Exported to userProfiles.json.

I then use the jq command line JSON processor to analyze the data and find issues, for example houses without creator:

$ jq -r 'with_entries(select(.value.createdBy == null))' houses.json

If it helps you and you have suggestions for improvements, PRs are always welcome! :)

Upvotes: 0

Neal Bayless
Neal Bayless

Reputation: 1

firestore-export-import has worked well for me, but here's a smaller version that I use when the structure of the firebase docs is simple.

const fs = require('fs')
const {initializeApp} = require('firebase/app')
const {getFirestore, collection, getDocs} = require('firebase/firestore')
const firebaseConfig = {
    // YOUR API KEY CONFIG
}
const app = initializeApp(firebaseConfig)
const db = getFirestore(app)

async function getAllData() {
    const docs = new Map()
    const profiles = await getDocs(collection(db, 'profiles'))
    profiles.forEach((doc) => {
        docs[doc.id] = doc.data()
    })
    const collections = new Map()
    collections['profiles'] = docs
    const fullDb = new Map()
    fullDb['__collections__'] = collections

    return fullDb
}

getAllData().then(fullDb => {
    const jsonString = JSON.stringify(fullDb, null, 2)
    fs.writeFile('./backup.json', jsonString, function (err) {
        if (err) {
            console.log(err)
        } else {
            console.log('export complete')
        }
    })
})

Upvotes: 0

MalanDev
MalanDev

Reputation: 13

Create a blank folder (Ex: firebaseImportExport ) and run npm init

Go to the source Firebase project -> Settings -> Service Accounts

Click on the Generate new private key button and rename the file as appConfig.json and put it in the firebaseImportExport folder

add packages

npm install firestore-export-import

create an index.js file.

paste the below code

const { initializeFirebaseApp } = require('firestore-export-import')

const  serviceAccount  =  require('./appConfig.json')

const  appName  =  '[DEFAULT]'

const firestore = initializeFirebaseApp(serviceAccount, appName)

const  fs  =  require('fs');

const { backup } = require('firestore-export-import')
//backup('collection name')

backup(firestore, 'collection-name').then((data) =>
{
    const  json  =  JSON.stringify(data);

    //where collection.json is your output file name.
    fs.writeFile('collection.json', json, 'utf8',()=>{

    console.log('done');

})

});

Upvotes: 0

Ayub Alam
Ayub Alam

Reputation: 1

I found an easier solution. There is a tool called Firefoo. It lists all the collection documents along with created users with multiple providers(email & password, phone number, google, facebook etc). You can export data in JSON & CSV along with that you can view the data in simplified format like Table, Tree, JSON.

Note:- You don't have to go through all the process for importing or exporting data from your firebase console.

Upvotes: 0

Andreas Klöber
Andreas Klöber

Reputation: 5920

Documents can also be downloaded as JSON via the REST API.

This is an example using curl in conjunction with the Cloud SDK to obtain an access token:

curl -H "Authorization: Bearer "$(gcloud auth print-access-token) \
  "https://firestore.googleapis.com/v1/projects/$PROJECT/databases/(default)/documents/$COLLECTION/$DOCUMENT"

Upvotes: 1

Gunar Gessner
Gunar Gessner

Reputation: 2631

Google made it harder than it needed to be, so the community found a workaround. If you have npm installed, you can do this:

Export

npx -p node-firestore-import-export firestore-export -a credentials.json -b backup.json

Import

npx -p node-firestore-import-export firestore-import -a credentials.json -b backup.json

Source

Upvotes: 57

Abhishek Ashware
Abhishek Ashware

Reputation: 47

for dumping json from your local to firestoreDB:

npx -p node-firestore-import-export firestore-import -a credentials.json -b backup.json

for downloading data from firestoreDB to your local:

npx -p node-firestore-import-export firestore-export -a credentials.json -b backup.json

to generate credentials.json, go to project settings -> service accounts -> generate a private key.

Upvotes: 4

Robin Manoli
Robin Manoli

Reputation: 2222

If someone wants a solution using Python 2 or 3.

Edit: note that this does not backup the rules

Fork it on https://github.com/RobinManoli/python-firebase-admin-firestore-backup

First install and setup Firebase Admin Python SDK: https://firebase.google.com/docs/admin/setup

Then install it in your python environment:

pip install firebase-admin

Install the Firestore module:

pip install google-cloud-core
pip install google-cloud-firestore

(from ImportError: Failed to import the Cloud Firestore library for Python)

Python Code

# -*- coding: UTF-8 -*-

import firebase_admin
from firebase_admin import credentials, firestore
import json

cred = credentials.Certificate('xxxxx-adminsdk-xxxxx-xxxxxxx.json') # from firebase project settings
default_app = firebase_admin.initialize_app(cred, {
    'databaseURL' : 'https://xxxxx.firebaseio.com'
})

db = firebase_admin.firestore.client()

# add your collections manually
collection_names = ['myFirstCollection', 'mySecondCollection']
collections = dict()
dict4json = dict()
n_documents = 0

for collection in collection_names:
    collections[collection] = db.collection(collection).get()
    dict4json[collection] = {}
    for document in collections[collection]:
        docdict = document.to_dict()
        dict4json[collection][document.id] = docdict
        n_documents += 1

jsonfromdict = json.dumps(dict4json)

path_filename = "/mypath/databases/firestore.json"
print "Downloaded %d collections, %d documents and now writing %d json characters to %s" % ( len(collection_names), n_documents, len(jsonfromdict), path_filename )
with open(path_filename, 'w') as the_file:
    the_file.write(jsonfromdict)

Upvotes: 11

albertkhang
albertkhang

Reputation: 701

It works for me.

I used Cloud Functions to export all data in Firestore to JSON format. The function that I was used:

exports.exportFirestore2Json = functions.https.onRequest((request, response) => {
    db.collection("data").get().then(function(querySnapshot) {
        const orders = [];
        var order = null

         querySnapshot.forEach(doc => {
             order = doc.data();
             orders.push(order);
         });

         response.send(JSON.stringify(orders))

         return true
    })
    .catch(function(error) {
        console.error("Error adding document: ", error);
        return false
    });
})

Then, go to https://your-project-id.cloudfunctions.net/exportFirestore2Json you will see something like this

enter image description here

Upvotes: 3

Nagibaba
Nagibaba

Reputation: 5358

Open any of your clientside firebase apps (React, Angular, etc.). Use this code anywhere to log console and copy

const products = await db
  .collection("collectionName")
  .where("time", ">", new Date("2020-09-01"))
  .get()


const json = JSON.stringify(products.docs.map((doc) => ({ ...doc.data() })))
console.log(json)

Upvotes: 0

Alegntaye Yilma
Alegntaye Yilma

Reputation: 11

  1. Create a blank folder (call it firebaseImportExport ) and run npm init
  2. Go to the source Firebase project -> Settings -> Service Accounts
  3. Click on the Generate new private key button and rename the file as source.json and put it in the firebaseImportExport folder
  4. Do the same (step 2 & 3) for the destination project and rename the file as destination.json
  5. Install the npm i firebase-admin npm package.
  6. Write the following code in the index.js
const firebase = require('firebase-admin');

var serviceAccountSource = require("./source.json");
var serviceAccountDestination = require("./destination.json");

const sourceAdmin = firebase.initializeApp({
    credential: firebase.credential.cert(serviceAccountSource),
    databaseURL: "https://**********.firebaseio.com" // replace with source
});

const destinationAdmin = firebase.initializeApp({
    credential: firebase.credential.cert(serviceAccountDestination),
    databaseURL: "https://$$$$$.firebaseio.com"
  }, "destination");

const collections = [ "books", "authors",   ...]; // replace with your collections

    var source = sourceAdmin.firestore();
    var destination = destinationAdmin.firestore();
   collections.forEach(colName => {
    source.collection(colName).get().then(function(querySnapshot) {
        querySnapshot.forEach(function(doc) {
            destination.collection(colName).doc(doc.id).set({...doc.data()});
        });
    });
   });

Upvotes: 1

Arihant Jain
Arihant Jain

Reputation: 847

Yes you can, you did not need to start billing in your firebase console. There is a great npm package https://www.npmjs.com/package/firestore-export-import with this you can export and import firestore collection and documents easily. Just follow some steps:

-Get your service account key Open Firebase console > Project settings > Service accounts > generate new private key

rename the downloaded file with serviceAccountKey.json

-Now create a new folder and index.js file.

-Paste you servicekey.json in this folder

-Now install this package

npm install firestore-export-import

OR

yarn add firestore-export-import

Exporting data from firebase

const { initializeApp} =  require('firestore-export-import')

const  serviceAccount  =  require('./serviceAccountKey.json')

const  appName  =  '[DEFAULT]'

initializeApp(serviceAccount, appName)

const  fs  =  require('fs');

const { backup } =  require('firestore-export-import')
//backup('collection name')

backup('users').then((data) =>
{
    const  json  =  JSON.stringify(data);

    //where collection.json is your output file name.
    fs.writeFile('collection.json', json, 'utf8',()=>{

    console.log('done');

})

});

Execute node index.js and you should see a new collection.json file with your collection and documents in it. If it looks a little messy pretty format it online with https://codebeautify.org/jsonviewer

This index.js was just a very basic configuration which exports the whole collection with everything in it, read their documentation you could do queries and much more!

Importing data to firebase

const { initializeApp,restore } =  require('firestore-export-import')

const  serviceAccount  =  require('./serviceAccountKey.json')
const  appName  =  '[DEFAULT]'

initializeApp(serviceAccount, appName)
restore('collection.json', {
//where refs is an array of key items
    refs: ['users'],
    //autoParseDates to parse dates if documents have timestamps
    autoParseDates: true,

    },()=>{

console.log('done');
})

After execution you should see your firestore populated with collection users!

Upvotes: 2

Hezy Ziv
Hezy Ziv

Reputation: 5560

There is an npm for firestore export / import

Project to export Goto -> project settings -> Service account -> Generate new private key -> save it as exportedDB.json

Project to import Goto -> project settings -> Service account -> Generate new private key -> save it as importedDB.json

run these 2 commands from the folder where u saved the files

Export: npx -p node-firestore-import-export firestore-export -a exportedDB.json -b backup.json

Import: npx -p node-firestore-import-export firestore-import -a importedDB.json -b backup.json

Upvotes: 8

Dale Nguyen
Dale Nguyen

Reputation: 1980

I just wrote a backup and restore for Firestore. You can have a try on my GitHub.

https://github.com/dalenguyen/firestore-backup-restore

Thanks,

Upvotes: 40

Dan McGrath
Dan McGrath

Reputation: 42018

There is not, you'd need to come up with your own process such as querying a collection and looping over everything.

Update

As of August 7th, 2018, we do have a managed export system that allows you to dump your data into a GCS bucket. While this isn't JSON, it is a format that is the same as Cloud Datastore uses, so BigQuery understands it. This means you can then import it into BigQuery.

Upvotes: 33

jloosli
jloosli

Reputation: 2581

I've written a tool that traverses the collections/documents of the database and exports everything into a single json file. Plus, it will import the same structure as well (helpful for cloning/moving Firestore databases). Since I've had a few colleagues use the code, I figured I would publish it as an NPM package. Feel free to try it and give some feedback.

https://www.npmjs.com/package/node-firestore-import-export

Upvotes: 16

willhlaw
willhlaw

Reputation: 39

Firestore is still early in its development so please check the docs on backups for any information pertaining to Firestore.

I found this npm package, node-firestore-backup, to be easy and useful.

Note that the --accountCredentials path/to/credentials/file.json is referring to a service account key json file that you can get by following instructions from https://developers.google.com/identity/protocols/application-default-credentials.

  1. Go to the API Console Credentials page.
  2. From the project drop-down, select your project.
  3. On the Credentials page, select the Create credentials drop-down, then select Service account key.
  4. From the Service account drop-down, select an existing service account or create a new one.
  5. For Key type, select the JSON key option, then select Create. The file automatically downloads to your computer.
  6. Put the *.json file you just downloaded in a directory of your choosing. This directory must be private (you can't let anyone get access to this), but accessible to your web server code.

Upvotes: 3

Related Questions