CODEWITHSUNDEEP
dockerkubernetesgoogle-kubernetes-engine
dekauliya
dekauliya

Reputation: 1433

How to separate application and data syncing implementations in Kubernetes?

I want to build an application/service which uses a static application data which can get updated over time. Currently, I implemented this by having both the application and the data within the same container. However, this requires redeployment when either application or data change. I want to separate the app and data volume implementation so that I can update the application and the data independently, meaning I won't have to rebuild the application layer when the application data is updated and vice versa.

Here are the characteristics of the Application Data and its Usage:

So here, we are interested in sync-ing the data in cloud storage to the volume in Kubernetes deployment. What's the best way to achieve this objective in Kubernetes?

I have several options in mind:

  1. Using one app container in one deployment, in which the app will also include the logic for data loading and update which pulls data from cloud storage to the container --> simple but tightly coupled with the storage read-write implementation

  2. Using the cloud store directly from the app --> this doesn't require container volume, but I was concerned with the huge file size because the app is an interactive service which requires a quick response

  3. Using two containers in one deployment sharing the same volume --> allow great flexibility for the storage read-write implementation

    • one container for application service reading from the shared volume
    • one container for updating data and listening to update data request which writes data to the shared volume --> this process will pull data from cloud storage to the shared volume

  4. Using one container with a Persistent Disk

    • an external process which writes to the persistent disk (not sure how to do this yet with cloud storage/file objects, need to find a way to sync gcs to persistent disk)
    • one container for application service which reads from the mounted volume

  5. Using Fuse mounts

    • an external process which writes to cloud storage
    • a container which uses fuse mounts

I am currently leaning towards option 3, but I am not sure if it's the common practice of achieving my objective. Please let me know if you have better solutions.

Upvotes: 1

Views: 392

Answers (2)

Chris Madden
Chris Madden

Reputation: 2650

Option 5. might now be more interesting because in the meantime Google Cloud released their own GCS Fuse driver including GKE specific integration. If you don't want to rewrite your application to use GCS APIs this could be a drop in replacement which fits well for a majority read workload like yours. It supports ranged reads so if your application only reads a small piece of a large file only the small piece will be fetched.

Performance benchmarks are available for you to verify if the latency is acceptable for your workload needs.

Upvotes: 0

Rico
Rico

Reputation: 61521

Yes. 3. is the most common option but make sure you use an initContainer to copy the data from your cloud storage to a local volume. That local volume could be any of the types supported by Kubernetes.

Upvotes: 1

Related Questions