Why is Google Compute Engine not running my container?

Question

I can do this successfully:

• Bundle my app into a docker image
• Build this image into a container using Google Cloud Build upon push to master
  • (This container is stored in the registry at, for example, gcr.io/my-project/my-container)
• Deply this container to the web using Google Cloud Run
• Visit the Cloud Run url and see my website

I am now trying more sophisticated builds and I think the next step is to use Google Compute Engine.

To start, I am simply trying to deploy a single instance of the same app that I deployed to Cloud Run:

• Navigate to Compute Engine > VM Instances
• Enter basics like instance name
• Enter my container location under "Container Image": gcr.io/my-project/my-container
  • (As an aside, I find it suspect that the interface does not offer a selector for your existing Container Registry items here.)
• Select "Allow HTTP Traffic" and "Allow HTTPS Traffic"
• Click "Create"

GCE takes a minute to create it, and then it shows the green checkmark and the instance name, and "External IP: 35.238.xxx.xxx". I visit that URL in my browser and get... "35.238.xxx.xxx refused to connect."

To inspect, I go back to the GCE page and select "SSH > Open in browser window" next to my instance, which opens a type of cloud terminal to the machine.

In this terminal window, type ps and see that no processes are running. The container Dockerfile ends with CMD yarn start:prod, so I guess that's not happening here.

Further, I ls here and there and navigate around, and see that there is no /app directory from my Dockerfile's WORKDIR /app command. It seems like not only did my app not boot, but was the container not copied to the VM instance?

What am I doing wrong?

Answer 1

In my case, the service account that I was using did not have access to the artifact registry. When the instance started up it resulted in a silent failure but when I gave access to the SA, the image was pulled just fine.

Answer 2

I also had the problem, the instance was running, but could not pull my container.

Error: Failed to start container: Error response from daemon: {"message":"unautho rized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"

I had to add some extra scope to the yaml file : https://www.googleapis.com/auth/source.full_control

steps:
- name: gcr.io/cloud-builders/docker
  args: ['build', '-t', 'gcr.io/local-xxxxxxxxxxxxxx/apptraining', '.']

- name: 'gcr.io/cloud-builders/docker'
  args: ["push", "gcr.io/local-xxxxxxxxxxxxxx/apptraining"]

- name: 'gcr.io/cloud-builders/gcloud'
  args: ['compute', 'instances', 'create-with-container', 'instanceapptraining', '--machine-type=n1-standard-1', '--scopes=https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/trace.append,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/bigquery,https://www.googleapis.com/auth/datastore,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/trace.append,https://www.googleapis.com/auth/source.full_control,https://www.googleapis.com/auth/source.read_only,https://www.googleapis.com/auth/compute.readonly','--zone=us-central1-a', '--preemptible', '--container-image=gcr.io/local-xxxxxxxxxxxxxx/apptraining:latest']

Answer 3

Check whether your VM has an external IP address.

If it doesn't, the VM might not have network access to the public repository and even to the Google Container Registry (gcr.io) and the docker container doesn't start silently.

Answer 4

For anyone having this issue. I faced the same problem and couldn't figure it out.

Reading Serhii's answer give me the clue. I believe as of today (Jan 2021) the GCP Console UI is a bit unhelpful. It appears that if you type in a container name when creating your VM but WITHOUT specifying a tag on the end, it doesn't complain nor assume a default such as 'latest', it just fails silently. Hence the VM but with no docker container running.

At least it this now works for me, hopefully this helps others.

Answer 5

I've decided to follow Deploying a container on a new VM instance again.

Please find my steps and commands below:

1. create a new VM that runs the Docker image gcr.io/cloud-marketplace/google/nginx1:latest with network tag http-server:

   $ gcloud compute instances create-with-container instance-3 --tags=http-server,https-server --container-image=gcr.io/cloud-marketplace/google/nginx1:latest
   Created [https://www.googleapis.com/compute/v1/projects/test-prj/zones/europe-west3-a/instances/instance-3].
   NAME        ZONE            MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
   instance-3  europe-west3-a  n1-standard-1               10.156.0.30  35.XXX.111.XXX  RUNNING
   

2. create a new firewall rule:

   $ gcloud compute firewall-rules create default-allow-http --direction=INGRESS --priority=1000 --network=default --action=ALLOW --rules=tcp:80 --source-ranges=0.0.0.0/0 --target-tags=http-server
   Creating firewall...⠹
   Created [https://www.googleapis.com/compute/v1/projects/test-prj/global/firewalls/default-allow-http].                   
   Creating firewall...done.                                                                                                                            
   NAME                NETWORK  DIRECTION  PRIORITY  ALLOW   DENY  DISABLED
   default-allow-http  default  INGRESS    1000      tcp:80        False
   

3. check current firewall rules:

   $ nmap -Pn 35.XXX.111.XXX
   Starting Nmap 7.70 ( https://nmap.org ) at 2020-04-02 12:04 CEST
   PORT   STATE SERVICE
   ...
   80/tcp open  http
   

4. check if NGINX is running in the container:

   $ curl -I http://35.XXX.111.XXX
   HTTP/1.1 200 OK
   Server: nginx/1.16.1
   ...
   
   $ curl http://35.XXX.111.XXX 
   ...
   <h1>Welcome to nginx!</h1>
   ... 
   

also via web browser at http://35.XXX.111.XXX

5. check status of the container:

   $ gcloud compute ssh instance-3
   ...
   instance-3 ~ $ docker ps
   CONTAINER ID        IMAGE                                                                COMMAND                  CREATED             STATUS              PORTS               NAMES
   ...
   a657c8871239        gcr.io/cloud-marketplace/google/nginx1:latest                        "/usr/local/bin/dock…"   14 minutes ago      Up 14 minutes                           klt-instance-3-uwtu
   

6. attach to the container and run curl http://35.XXX.111.XXX in the separate terminal:

   instance-3 ~ $ docker attach a657c8871239
   YY.YY.43.203 - - [02/Apr/2020:10:18:06 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.0" "-"
   YY.YY.43.203 - - [02/Apr/2020:10:18:07 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.0" "-"
   

I found no errors while following documentation.

To solve your issue:

1. Compare your steps and commands to mine.
2. Run test Docker image by following documentation on your project.
3. Try to replicate steps from documentation with your custom image.

If you still have issue - update your question with all your steps, commands and outputs.