Where is the headless chrome browser in a Google App Engine

Question

I am looking for the location of an executable in an Google App Engine (standard environment).

The reason is that I am trying to use pyppeteer for some work but pyppeteer always downloads chromium into a custom folder and then exits.

I saw that puppeteer works on GAE and I hope therefore that I can convince pyppeteer to run as well.

Here are the Google App Engine logs:

2019-07-23 17:39:33 default[test]  [W:pyppeteer.chromium_downloader] start chromium download.
2019-07-23 17:39:33 default[test]  Download may take a few minutes.
2019-07-23 17:39:33 default[test]
2019-07-23 17:39:34 default[test]    0%|          | 0/106826418 [00:00<?, ?it/s]
2019-07-23 17:39:34 default[test]    2%|▏         | 1925120/106826418 [00:00<00:07, 13602613.39it/s]
2019-07-23 17:39:34 default[test]    4%|▍         | 4392960/106826418 [00:00<00:06, 15716861.03it/s]
2019-07-23 17:39:34 default[test]    6%|▌         | 6256640/106826418 [00:00<00:06, 16489218.92it/s]
2019-07-23 17:39:34 default[test]    8%|▊         | 8407040/106826418 [00:00<00:05, 17728112.85it/s]
2019-07-23 17:39:34 default[test]   11%|█         | 11683840/106826418 [00:00<00:04, 20557603.00it/s]
2019-07-23 17:39:34 default[test]   13%|█▎        | 14356480/106826418 [00:00<00:04, 22086461.51it/s]
2019-07-23 17:39:34 default[test]   16%|█▌        | 17008640/106826418 [00:00<00:03, 23246105.82it/s]
2019-07-23 17:39:34 default[test]   18%|█▊        | 19374080/106826418 [00:00<00:03, 23225442.25it/s]
2019-07-23 17:39:35 default[test]   20%|██        | 21729280/106826418 [00:00<00:03, 23203081.94it/s]
2019-07-23 17:39:35 default[test]   24%|██▍       | 25681920/106826418 [00:01<00:03, 26483368.08it/s]
2019-07-23 17:39:35 default[test]   27%|██▋       | 28508160/106826418 [00:01<00:02, 26759076.64it/s]
2019-07-23 17:39:35 default[test]   29%|██▉       | 31313920/106826418 [00:01<00:02, 26995282.49it/s]
2019-07-23 17:39:35 default[test]   32%|███▏      | 34375680/106826418 [00:01<00:02, 27987817.70it/s]
2019-07-23 17:39:35 default[test]   35%|███▍      | 37253120/106826418 [00:01<00:02, 28005026.46it/s]
2019-07-23 17:39:35 default[test]   38%|███▊      | 40212480/106826418 [00:01<00:02, 28458335.10it/s]
2019-07-23 17:39:35 default[test]   40%|████      | 43100160/106826418 [00:01<00:02, 28379615.37it/s]
2019-07-23 17:39:35 default[test]   43%|████▎     | 45967360/106826418 [00:01<00:02, 22218027.73it/s]
2019-07-23 17:39:36 default[test]   45%|████▌     | 48414720/106826418 [00:01<00:02, 22336538.22it/s]
2019-07-23 17:39:36 default[test]   48%|████▊     | 51435520/106826418 [00:02<00:02, 24224996.22it/s]
2019-07-23 17:39:36 default[test]   51%|█████▏    | 54763520/106826418 [00:02<00:01, 26376309.32it/s]
2019-07-23 17:39:36 default[test]   54%|█████▍    | 57569280/106826418 [00:02<00:02, 21505161.66it/s]
2019-07-23 17:39:36 default[test]   56%|█████▌    | 60078080/106826418 [00:02<00:02, 22466550.18it/s]
2019-07-23 17:39:36 default[test]   59%|█████▉    | 63027200/106826418 [00:02<00:01, 24192591.42it/s]
2019-07-23 17:39:36 default[test]   61%|██████▏   | 65628160/106826418 [00:02<00:01, 24070010.24it/s]
2019-07-23 17:39:36 default[test]   64%|██████▍   | 68556800/106826418 [00:02<00:01, 25428249.96it/s]
2019-07-23 17:39:36 default[test]   68%|██████▊   | 72529920/106826418 [00:02<00:01, 28506739.68it/s]
2019-07-23 17:39:37 default[test]   71%|███████   | 75581440/106826418 [00:02<00:01, 28986836.20it/s]
2019-07-23 17:39:37 default[test]   74%|███████▎  | 78622720/106826418 [00:03<00:00, 29210576.39it/s]
2019-07-23 17:39:37 default[test]   76%|███████▋  | 81643520/106826418 [00:03<00:01, 22926790.84it/s]
2019-07-23 17:39:37 default[test]   79%|███████▉  | 84224000/106826418 [00:03<00:00, 23612840.24it/s]
2019-07-23 17:39:37 default[test]   82%|████████▏ | 87193600/106826418 [00:03<00:00, 25157952.55it/s]
2019-07-23 17:39:37 default[test]   84%|████████▍ | 89937920/106826418 [00:03<00:00, 25797481.66it/s]
2019-07-23 17:39:37 default[test]   87%|████████▋ | 92672000/106826418 [00:03<00:00, 26235841.07it/s]
2019-07-23 17:39:37 default[test]   89%|████████▉ | 95488000/106826418 [00:03<00:00, 26776710.45it/s]
2019-07-23 17:39:37 default[test]   92%|█████████▏| 98232320/106826418 [00:03<00:00, 26824171.60it/s]
2019-07-23 17:39:38 default[test]   95%|█████████▍| 100966400/106826418 [00:03<00:00, 26803548.31it/s]
2019-07-23 17:39:38 default[test]   97%|█████████▋| 103925760/106826418 [00:04<00:00, 27583062.30it/s]
2019-07-23 17:39:38 default[test]  100%|██████████| 106826418/106826418 [00:04<00:00, 25706775.94it/s]
2019-07-23 17:39:38 default[test]  [W:pyppeteer.chromium_downloader]
2019-07-23 17:39:38 default[test]  chromium download done.
2019-07-23 17:39:44 default[test]  [W:pyppeteer.chromium_downloader] chromium extracted to: /root/.local/share/pyppeteer/local-chromium/575458
2019-07-23 17:39:46 default[test]  [I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:22168/devtools/browser/c820f895-a90a-4242-a3fa-35dfda06be3f
2019-07-23 17:39:48 default[test]  [2019-07-23 17:39:48 +0000] [9] [INFO] Handling signal: term
2019-07-23 17:39:48 default[test]  [2019-07-23 17:39:48 +0000] [17] [INFO] Worker exiting (pid: 17)```

Answer 1

You can actually run Pyppeteer on Google App Engine Flex. You just need to define a Dockerfile and point to the image container gcr.io/google_appengine/python which is GCR's standard Python image.

Then you'll want to add all the linux libraries that allows Chromium to run.

Here is the docker file:

FROM gcr.io/google_appengine/python
RUN apt-get update && apt-get -y install  gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget

RUN virtualenv /env
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH

RUN mkdir /your/app/path
ADD . /your/app/path
WORKDIR /your/app/path
ADD requirements.txt /your/app/path/requirements.txt
RUN pip3 install -r /your/app/path/requirements.txt
EXPOSE 8080
ENTRYPOINT ["gunicorn", "-b", ":8080", "main:app"]

Then in your python file set the flag

args = ['--no-sandbox']
browser = await launch(args=args)
page = await browser.newPage()

So Chromium can run as root.

You can check out my full walk through and explanation here:

Answer 2

Pyppeteer is downloading the chromium executable to the local filesystem which is not writeable [1], that is why when it tries to access the file, it is no longer there and it exits. You could download chromium to a storage bucket on Google Cloud Storage and then read the file from there.

[1] Reading and Writing Files

In App Engine, the local filesystem that your application is deployed to is not writeable. This behavior ensures the security and scalability of your application.

However, if the application needs to write and read files at runtime, App Engine provides a built-in Google Cloud Storage stream wrapper that allows you to use many of the standard PHP filesystem functions to read and write files in an App Engine PHP app.