Sharing code between flexible environment modules in a GAE project

Question

I'm structuring my GAE (flex) project as a number of services:

- my-project/
  - services/
    - service_1/
      - service_1.yaml
    - service_2/
      - service_2.yaml

I would now like to share some code between the services. Ideally, I'd like to have a my-project/lib/ and symlink this to the services:

• my-project/services/service_1/lib -> ../../lib
• my-project/services/service_2/lib -> ../../lib

But gcloud app deploy (which I'm running from a service directory, where the yaml file is) ignores the symlink (I checked the tgz file before it got uploaded got GAE for deployment and lib/ isn't there).

So, is there any way to share the lib/ code (apart from pulling it into it's own library and installing with pip)?

Answer 1

Copying my answer from a similar question. Multiple services with different dockerfiles on GAE Flexible

tl;dr: build a separate docker image, push it to GCR, deploy using that image.

Specify a custom runtime. Build the image locally, tag it, and push it to Google Container Registry (GCR) then, deploy your service, specifying a custom service file, and specifying the remote image on GCR using the --image-url option.

Here's an example that accomplishes different entrypoints in 2 services that share the same code: ...this is assuming that the "flex" and not "standard" app engine offering is being used.

lets say you have a project called my-proj with a default service that is not important and a second service called queue-processor which is using much of the same code from the same directory. Create a separate dockerfile for it called QueueProcessorDockerfile and a separate app.yaml called queue-processor-app.yaml to tell google app engine what i want to happen.

QueueProcessorDockerfile

FROM node:10
# Create app directory
WORKDIR /usr/src/app
COPY package.json ./
COPY yarn.lock ./
RUN npm install -g yarn
RUN yarn
# Bundle app source
COPY . .
CMD [ "yarn", "process-queue" ]

*of course i have a "process-queue" script in my package.json queue-processor-app.yaml

runtime: custom
env: flex
... other stuff...
...

1. build and tag the docker image Check out googles guide here -> https://cloud.google.com/container-registry/docs/pushing-and-pulling docker build -t eu.gcr.io/my-proj/queue-processor -f QueueProcessorDockerfile .
2. push it to GCR docker push eu.gcr.io/my-proj/queue-processor
3. deploy the service, specifying which yaml config file google should use, as well as the image url you have pushed gcloud app deploy queue-processor-app.yaml --image-url eu.gcr.io/my-proj/queue-processor

Answer 2

I came up with another solution that seems to the best of a bunch of bad alternatives... This is what I ruled out:

• hard links -- not supported by GIT
• git submodules or subtrees -- too complicated
• splitting the shared code to an different project and adding to requirements.txt -- also too complicated

My solution was to create this two line bash script:

rm -rf deploy_dir
cp -RLp code_dir deploy_dir

It copies all my code to a new directory, and in doing the copying, all sym links get replaced with the files they linked to. I then deploy from deploy_dir

Answer 3

The method of sharing code across services using symlinks might not be applicable to the flex environment, which is docker-based. This suspicion is based on the last table row from Troubleshooting Custom Runtimes:

Issue

• Some files are not included in my application.

Solutions

• Docker does not follow symlinks when added by your source tree or your packaging system to the base image, so any source files outside of your applications directory that are referenced by symlinks in your source tree will not be copied into your application.

Also - dependencies in the flex environment are supposed to be based on a requirements.txt file used to build the dockerfile at deployment time, not on vendoring in libraries by pip installing them into the lib dir which is the standard environment way.