Google Firebase Firestore JSON Export

Exports a Google Firestore database recursively including all collections and subcollections into plain JSON files.

Uses the built-in recursive() flag of the Query class: https://cloud.google.com/python/docs/reference/firestore/latest/query#recursive

A single recursive query over a large collection times out on the server. Each collection is therefore split into ranges of 25 top-level documents (including all of their subcollections), which are paged through and exported in parallel.

Allows you to analyze the data in your Firestore database locally without being constrained to Firestore or BigQuery limitations. For example:

• Finding incomplete documents
• Finding broken document references
• Finding documents without fields
• Querying subcollections of collections without documents

You can use custom scripts or the powerful jq command-line JSON processor to perform complex lookups, such as this example, or the following, which extracts all documents that do not have the key 'createdBy':

jq -r 'with_entries(select(.value.createdBy == null))' collection.json

⚠️ Warning: A complete export causes 1 read operation for every document in your Firestore database. Depending on the size of your database, this might induce costs for database reads.

Requirements

• Python 3

Setup and Usage

1. Install firebase-admin via pip

   $ pip3 install firebase-admin

2. Run export to JSON.

   $ python3 main.py

   To export only some collections, pass their names:

   $ python3 main.py meals dayplans

Todos

• Serialize TimestampWithNanoseconds (instead of converting to string) https://code.luasoftware.com/tutorials/google-cloud-firestore/python-firestore-query-documents-to-json

• Serialize document references (e.g. as document path instead of address)

  "ref": "<google.cloud.firestore_v1.document.DocumentReference object at 0x111ef1c50>"
  

• Import into local Firestore emulator

• Allow to export as line-delimited JSON (LDJSON) to allow streaming and avoid loading the whole db into memory when importing.