CODEWITHSUNDEEP
node.jsdockernginxgoogle-cloud-platformpuppeteer
Moshe
Moshe

Reputation: 2684

Deploying NodeJS, Puppeteer, and Angular on Google Cloud

I have the following setup:

Angular Front-end
MongoDb database
NodeJS/Express backend for API (+ puppeteer)

steps to deployment:
1. ng build -prod --> creates dist in /server
2. gcloud app deploy (Node.js + Express + Puppeteer + Dockerfile config*)

Because puppeteer is basically chrome; some libraries which aren't given to me, had to be uploaded. No problem, Docker to the rescue.

FROM node:8-slim

MAINTAINER Eric Bidelman <ebidel@>


RUN apt-get update && apt-get install -y wget --no-install-recommends \
    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-unstable \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get purge --auto-remove -y curl \
    && rm -rf /src/*.deb

COPY . /app/
WORKDIR app

# Install deps for server.
RUN yarn


ARG CACHEBUST=1
RUN yarn add puppeteer

RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
    && mkdir -p /home/pptruser/Downloads \
    && chown -R pptruser:pptruser /home/pptruser \
    && chown -R pptruser:pptruser /app

USER pptruser

EXPOSE 8080

CMD ["yarn", "start"]

this was taken from "try-puppeteer" github repo (https://github.com/ebidel/try-puppeteer)

Now the front-end is a private site, max 2 people on. How do I deploy Angular in a way where NodeJS doesn't have to solve it? Someone suggested "Google Cloud Bucket for $0.09/mo" which is cool.

Now the MongoDB instance is using google cloud compute engine. The database has a collection of 400 documents, which are at the size of 11kb each. Still on free resources here.

Functionality on the site: For all (filtered) documents --> do action which requires us to load pupeteer().then( res.send(results))

This means that there is an average of 200 API requests from the frontend being handled in one blow. Even though NodeJS handles this well (one by one, synchronously) I was thinking about adding more "NodeJS-clones", and maybe I can handle simultaneous requests this way?

--This makes me wonder, if I have a NodeJS server for 300 clients at a time; how are the requests being handled? How don't I feel a "stoppage" between requests? Is there a queue that goes 1 by 1; or is this a limitation on AJAX?

In other words, how can I deploy this in a way that I can handle this waterfall of requests?

I've read up on "nginx" and "forever," but I'm more thinking along the lines of loading perhaps 10 instances of puppeteer (even from the same VM, like tabs or windows) where they can all work concurrently.

Here's where I'm stuck: I understand I can deploy node clones; but how would I do it with google cloud where I can scale through dividing and conquoring. (200 requests at 2 requests "per tick" is twice as fast as 200 requests at 1 request "per tick".)

Am I doing this the wrong way? Making 200 requests to the backend instead of making one request with 200 objects -- another problem is the fear of timeout. Each request takes 40 seconds to complete.

App.yaml for "default" (be + fe deployed together):

runtime: custom
env: flex
service: default

automatic_scaling:
  max_num_instances: 1

resources:
  cpu: 1
  memory_gb: 1
  disk_size_gb: 10

skip_files:
- node_modules/
- test*.js
- ^(.*/)?.*\.md$

Note: the exact app.yaml from the repo drove up my costs to $250/13 days. I need to find a cheaper way to execute this program.

Upvotes: 0

Views: 1319

Answers (1)

browserless
browserless

Reputation: 2132

I'm not entirely sure how billing works on Google Cloud. If it's message throughput then running headless Chrome is going to be expensive no matter what since it sends requests through WebSockets (meaning at a cost, likely).

This is something that browserless is designed bottom-up for: headless browsers in the cloud. I've also open-sourced the underlying docker images here if you're interested.

I think the bottom line is that many/most providers out there just really aren't well setup to handle binaries executing like a browser. Either their way of deploying makes it tough, or that their methodologies behind billing make it expensive. Believe me when I say that your issue regarding billing and transparency aren't the first I have encountered.

Upvotes: 1

Related Questions