How to delete all documents from Gcloud beta emulators firestore?

Question

I'm creating some trivial apps to learn Firestore.

I started the local Firestore Emulator with:

$ gcloud beta emulator firestore start

After starting the emulator, I ran tests with "go test"

I populated the Firestore with data and created a function that queried some of the records/documents added.

I deleted some of the Documents from my app but they continue to show up in Queries.

I tried:

• stoping with ctrl-c and ctrl d

• $ gcloud beta emulator firestore stop

• restarted my Macbook but the Documents persist.

I don't understand how the datastore is persisting after restarting the computer, I'm guessing that the data is stored in a JSON file or something like that.

I searched but was not able to find any documentation on the emulator.

Am I supposed to start the emulator and then run tests against the emulated Firestore?

How do I flush the Firestore?

Answer 1

This is not documented, but here are the endpoints Emulators UI uses. My examples use fetch to send requests and assume you use default ports.

1. Remove all data from Firestore emulator:

fetch('http://127.0.0.1:8080/emulator/v1/projects/PROJECT_NAME/databases/(default)/documents', { method: 'DELETE' })

2. Remove all accounts from Auth emulator:

fetch('http://127.0.0.1:9099/emulator/v1/projects/PROJECT_NAME/accounts', { method: 'DELETE' })

3. Remove all files from Storage buckets is trickier, use this function:

async function clearStorage() {
  const baseURL = 'http://127.0.0.1:9199/';
  const { items: buckets } = await (await fetch(baseURL + 'b')).json();
  const deleteRequests = [];
  for (const { id } of buckets) {
    const bucketURL = `${baseURL}v0/b/${id}/o/`;
    const { items } = await (await fetch(bucketURL)).json();
    for (const { name } of items) {
      const deleteURL = bucketURL + encodeURIComponent(name);
      deleteRequests.push(fetch(deleteURL, { method: 'DELETE' }));
    }
  }
  return Promise.all(deleteRequests);
}

---

For the sake of giving a full example, here is the code for my jest setup:

const PROJECT_NAME = 'some-firebase-project' // replace with your project name

beforeEach(async () => {
  await clearAllEmulatorsData();
});

async function clearAllEmulatorsData() {
  // This is not officially documented approach and can eventually fail.
  return Promise.all([clearAuth(), clearFirestore(), clearStorage()]);
}

async function clearAuth() {
  const deleteURL = `http://127.0.0.1:9099/emulator/v1/projects/${PROJECT_NAME}/accounts`;
  return fetch(deleteURL, { method: 'DELETE' });
}

async function clearFirestore() {
  const deleteURL = `http://127.0.0.1:8080/emulator/v1/projects/${PROJECT_NAME}/databases/(default)/documents`;
  return fetch(deleteURL, { method: 'DELETE' });
}

async function clearStorage() {
  const baseURL = 'http://127.0.0.1:9199/';
  const { items: buckets } = await (await fetch(baseURL + 'b')).json();
  const deleteRequests = [];
  for (const { id } of buckets) {
    const bucketURL = `${baseURL}v0/b/${id}/o/`;
    const { items } = await (await fetch(bucketURL)).json();
    for (const { name } of items) {
      const deleteURL = bucketURL + encodeURIComponent(name);
      deleteRequests.push(fetch(deleteURL, { method: 'DELETE' }));
    }
  }
  return Promise.all(deleteRequests);
}

Answer 2

Since you're using Go, here's a little test helper I implemented that helps with starting the emulator, waiting for it to come up, purging existing data, initializing a client, and shutting down the operator after it is completed.

It uses the technique in Juan's answer (which you should mark as the answer).

To use this utility, you need to just say:

client := startFirestoreEmulator(t)

Source code:

// Copyright 2021 Ahmet Alp Balkan
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package firestoretestutil contains test utilities for starting a firestore
// emulator locally for unit tests.
package firestoretestutil

import (
    "bytes"
    "context"
    "fmt"
    "net"
    "net/http"
    "os"
    "os/exec"
    "sync"
    "testing"
    "time"

    firestore "cloud.google.com/go/firestore"
)

const firestoreEmulatorProj = "dummy-emulator-firestore-project"

// cBuffer is a buffer safe for concurrent use.
type cBuffer struct {
    b bytes.Buffer
    sync.Mutex
}

func (c *cBuffer) Write(p []byte) (n int, err error) {
    c.Lock()
    defer c.Unlock()
    return c.b.Write(p)
}

func StartEmulator(t *testing.T, ctx context.Context) *firestore.Client {
    t.Helper()
    port := "8010"
    addr := "localhost:" + port
    ctx, cancel := context.WithCancel(ctx)
    t.Cleanup(func() {
        t.Log("shutting down firestore operator")
        cancel()
    })

    // TODO investigate why there are still java processes hanging around
    // despite we kill the exec'd command, suspecting /bin/bash wrapper that gcloud
    // applies around the java process.
    cmd := exec.CommandContext(ctx, "gcloud", "beta", "emulators", "firestore", "start", "--host-port="+addr)
    out := &cBuffer{b: bytes.Buffer{}}
    cmd.Stderr, cmd.Stdout = out, out
    if err := cmd.Start(); err != nil {
        t.Fatalf("failed to start firestore emulator: %v -- out:%s", err, out.b.String())
    }
    dialCtx, clean := context.WithTimeout(ctx, time.Second*10)
    defer clean()
    var connected bool
    for !connected {
        select {
        case <-dialCtx.Done():
            t.Fatalf("emulator did not come up timely: %v -- output: %s", dialCtx.Err(), out.b.String())
        default:
            c, err := (&net.Dialer{Timeout: time.Millisecond * 200}).DialContext(ctx, "tcp", addr)
            if err == nil {
                c.Close()
                t.Log("firestore emulator started")
                connected = true
                break
            }
            time.Sleep(time.Millisecond * 200) //before retrying
        }
    }
    os.Setenv("FIRESTORE_EMULATOR_HOST", addr)
    cl, err := firestore.NewClient(ctx, firestoreEmulatorProj)
    if err != nil {
        t.Fatal(err)
    }
    os.Unsetenv("FIRESTORE_EMULATOR_HOST")
    truncateDB(t, addr)
    return cl
}

func truncateDB(t *testing.T, addr string) {
    t.Helper()
    // technique adopted from https://stackoverflow.com/a/58866194/54929
    req, _ := http.NewRequest(http.MethodDelete, fmt.Sprintf("http://%s/emulator/v1/projects/%s/databases/(default)/documents",
        addr, firestoreEmulatorProj), nil)
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        t.Fatal(err)
    }
    if resp.StatusCode != http.StatusOK {
        t.Fatalf("failed to clear db: %v", resp.Status)
    }
}

Answer 3

The emulator supports an endpoint to clear the database (docs):

curl -v -X DELETE "http://localhost:PORT/emulator/v1/projects/PROJECT_NAME/databases/(default)/documents"

Fill in the PORT and PROJECT_NAME.

Answer 4

You can use this:

module.exports.teardown = async () => {
  Promise.all(firebase.apps().map(app => app.delete()));
};

Now, each time you call teardown, you will delete all data from the Firestore emulator.