CODEWITHSUNDEEP
gun
user2672083
user2672083

Reputation: 375

gun db storage model for large centrally stored data, tiny collaborative clients

Use Case:

Say I wanted to create a realtime-collaborative document editing system.

In this scenario many users could create and collaborate on many documents.

Due to client-device constraints, it's not possible for any client to keep a replica of all documents, only just a handful.

There needs to be central storage server where all documents always live, and this server is always backed up.

Each client can 'subscribe' to any document, and all clients subscribed can see realtime changes of all other clients subscribed/editing the same document.

Questions:

Since each client can't store all documents, there needs to be a way to remove the replicas of 'old' documents from the client, without deleting the document from the central store, ideally based on an automatic least-recently-used approach. How is this handled in gun?

In gun, how can a document be deleted from the central store, so it's then effectively permanently removed from, and no longer accessible to, all clients?

When a document is deleted from the central store, how is the physical storage space ever actually reclaimed for later use?

Upvotes: 1

Views: 1184

Answers (1)

marknadal
marknadal

Reputation: 7624

Great questions, @user2672083 . Here is the current lay out:

  1. Collaborative realtime document editing is possible with gun. Here is a quick prototype I recorded a long time ago, however there are no full pre-built examples/implementations yet.

  2. Not all data is stored on every client by default. The browser only keeps the data it requests/gets/subscribes to.

  3. The default server already acts as a backup. I recommend using the S3 storage adapter, because then you do not have to worry about running out of disk space.

  4. Removing old replicas. Currently, if I want the server to act as a central "master", I just put a localStorage.clear() at the top of my browser code. This will force the browser to have to always load the latest from the server. This is not ideal though, an LRU specific feature is coming soon according to the roadmap.

  5. Permanently removing data and reclaiming space. While this should be easy for a central setup, because gun is P2P by default, it uses a technique called tombstoning to delete data. Given a lot of requests (like yours) for LRU/TTL/GC/deleting, there will be better support for this in the future. Currently, you have to use a mix of rm data.json, localStorage.clear() and 30 day lifecycles on S3 to get this to work. This will be more integrated/easier in the future.

Now a question for you: What are you working on, and how can I help? Many of the things you asked about are possible (with some work) now, but slated to be the focus of the next version of gun - I'd love to get your feedback as we build this out.

All peers reply to requests for data (#2), meaning that localStorage and the server will both reply. Because localStorage is physically closer to a user, it will reply first/fastest and then replies from the server will be merged. GUN does not try each peer "in sequence" doing try/catch cascades, GUN replies from all peers in parallel.

GUN has swappable storage and transport interfaces, so yes, it is easy to build other layers on top or into it.

Upvotes: 3

Related Questions