CODEWITHSUNDEEP
google-app-engine
Alex Conrad
Alex Conrad

Reputation: 359

How to clone a git repository from Google App Engine?

I host an app on Google App Engine (GAE). Upon an incoming request, I need to add a task to a push queue which in turn triggers a worker (fires another request to my app) to clone a git repository in a temporary to analyze it (git repo can be removed after the worker is done).

I consider using a Python git library to clone the git repository programmatically (e.g. GitPython) but given the file system limitations (no writes) I'm unsure how to achieve this. Is there some kind of volatile / in-memory filesystem available (e.g. ramdisk) that I can use as a scratch directory to do my work?

Also, I stumbled upon MemoryFS but I haven't tried it out yet and don't know if that would work on GAE.

Upvotes: 2

Views: 1013

Answers (1)

Dan Cornilescu
Dan Cornilescu

Reputation: 39814

According to the documentation, the GAE Managed VMs allow running plain GAE app code (python, java and go) in less restrictive sandboxes. In particular a managed VM can be configured to have a real local filesystem:

You can choose the hosting environment (sandboxed or managed VM) separately for each module in your application. This means an application can contain a mix of modules that use different hosting environments. For instance, you might use the sandbox environment for your client-facing frontend, and use a module running in a managed VM for backend processing. Note that every instance of any specific module will run in the same type of environment.

The following table summarizes the differences between the two App Engine hosting enviroments:

Feature                   App Engine sandbox          Managed VM
...
Writing to local disk     No                          Yes, ephemeral (disk initialized on each VM startup)
...
Pricing                   Based on Instance hours     While in Beta, based on Compute Engine Pricing for each VM. Pricing will change in the future.

So you should be able to have a separate app module performing filesystem-related jobs configured to run on a vm.

Upvotes: 1

Related Questions