How can I export my Firestore database from a Google Storage Bucket into a Json file

Question

Here's the situation: I have a Firestore Database. I download it on a daily basis to a Google Cloud Storage Bucket as a backup. If I want to download it locally, I do it using this command gsutil -m cp -r gs://BUCKET_PATH "DESTINATION_PATH" and it works fine.

MY PROBLEM: the format of the Bucket I download is LevelDB (I think). On my machine, it looks like this :

For example, this is my /users collection in Firestore when I download it through Google Storage. In the folder, I have multiple binary files ("output-...") and a file ( here: "all_namespaces_kind_users") for metadata.

MY GOAL: I want to be able to read my database in a json file.

MY TRIES:

• I try to use this convertor : https://github.com/Venryx/firestore-leveldb-tools but it uses Python2 and some old google libraries. Using this convertor, I have to download the SDKs locally (see below).

#repoRoot = os.getcwd()
repoRoot = os.path.dirname(os.path.realpath(__file__))

# import google sdks
sys.path.append(os.path.join(repoRoot, 'SDKs/google_appengine'))
sys.path.append(os.path.join(repoRoot, 'SDKs/google-cloud-sdk/lib/third_party'))
from google.appengine.api.files import records
from google.appengine.datastore import entity_pb
from google.appengine.api import datastore

• Using the precedent convertor, I tried updating it to Python3, using these recommandations : https://github.com/Venryx/firestore-leveldb-tools/issues/1#issuecomment-674563473. I succeed to replace the first import from google.appengine.api.files import records but not the two others.
• For the two others : from google.appengine.datastore import entity_pb and from google.appengine.api import datastore, I saw this : https://pypi.org/project/protobuf-cloud-datastore-translator/ but I dont understand it fully.

MY QUESTION: How can I convert my Firestore Database (stored in a Google Bucket) to a json file ? Maybe someone has a complete different approach to propose ? Or should I stick to my way ? In that case, how can I finish to translate the convertor to Python3 using updated google librairies ?

BONUS QUESTION: How the h**l does not Google has a clean solution for that ? (or am I stupid?)

Answer 1

I have created a converter in Python 3 which could convert firestore export files into JSON files firestore-export-json. The package provides a simple CLI command to covert the file.

fs_to_json [path_to_source_dir] -d [path_to_destination]