CODEWITHSUNDEEP
google-compute-engine
treffner
treffner

Reputation: 121

Google cloud compute engine freeze

Today I found out that my vm instance in the google cloud had an issue.

I could not connect via gcloud compute ssh. The only solution was to shut the vm down and restart it.

I guess I found the right log file where the error happened.

/var/log/daemon.log

Nov 23 17:39:01 globo-de systemd[1]: Started Clean php session files.
Nov 23 18:09:01 globo-de systemd[1]: Starting Clean php session files...
Nov 23 18:09:03 globo-de systemd[1]: Started Clean php session files.
Nov 23 18:39:01 globo-de systemd[1]: Starting Clean php session files...
Nov 23 18:39:04 globo-de systemd[1]: Started Clean php session files.
Nov 23 18:56:49 globo-de google-ip-forwarding: WARNING Exception running ['ip', 'route', 'ls', 'table', 'local', 'type', 'local', 'scope', 'host', 'dev', 'eth0', 'proto', '66']. [Errno 12] Cannot allocate memory.
Nov 23 18:57:43 globo-de google-accounts: ERROR Exception calling the response handler. [Errno 12] Cannot allocate memory.#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/metadata_watcher.py", line 196, in WatchMetadata#012    handler(response)#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/accounts_daemon.py", line 259, in HandleAccounts#012    self.oslogin.UpdateOsLogin(enable=False)#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/oslogin_utils.py", line 79, in UpdateOsLogin#012    status = self._GetStatus()#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/oslogin_utils.py", line 60, in _GetStatus#012    retcode = self._RunOsLoginControl('status')#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/oslogin_utils.py", line 47, in _RunOsLoginControl#012    return subprocess.call([constants.OSLOGIN_CONTROL_SCRIPT, action])#012  File "/usr/lib/python2.7/subprocess.py", line 168, in call#012    return Popen(*popenargs, **kwargs).wait()#012  File "/usr/lib/python2.7/subprocess.py", line 390, in __init__#012    errread, errwrite)#012  File "/usr/lib/python2.7/subprocess.py", line 916, in _execute_child#012    self.pid = os.fork()#012OSError: [Errno 12] Cannot allocate memory
Nov 23 18:58:02 globo-de google-ip-forwarding: WARNING Exception running ['ip', 'route', 'ls', 'table', 'local', 'type', 'local', 'scope', 'host', 'dev', 'eth0', 'proto', '66']. [Errno 12] Cannot allocate memory.
Nov 23 18:59:35 globo-de google-ip-forwarding: WARNING Exception running ['ip', 'route', 'ls', 'table', 'local', 'type', 'local', 'scope', 'host', 'dev', 'eth0', 'proto', '66']. [Errno 12] Cannot allocate memory.
Nov 23 18:59:54 globo-de google-accounts: ERROR Exception calling the response handler. [Errno 12] Cannot allocate memory.#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/metadata_watcher.py", line 196, in WatchMetadata#012    handler(response)#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/accounts_daemon.py", line 259, in HandleAccounts#012    self.oslogin.UpdateOsLogin(enable=False)#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/oslogin_utils.py", line 79, in UpdateOsLogin#012    status = self._GetStatus()#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/oslogin_utils.py", line 60, in _GetStatus#012    retcode = self._RunOsLoginControl('status')#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/oslogin_utils.py", line 47, in _RunOsLoginControl#012    return subprocess.call([constants.OSLOGIN_CONTROL_SCRIPT, action])#012  File "/usr/lib/python2.7/subprocess.py", line 168, in call#012    return Popen(*popenargs, **kwargs).wait()#012  File "/usr/lib/python2.7/subprocess.py", line 390, in __init__#012    errread, errwrite)#012  File "/usr/lib/python2.7/subprocess.py", line 916, in _execute_child#012    self.pid = os.fork()#012OSError: [Errno 12] Cannot allocate memory
Nov 23 19:03:53 globo-de google-ip-forwarding: WARNING Exception running ['ip', 'route', 'ls', 'table', 'local', 'type', 'local', 'scope', 'host', 'dev', 'eth0', 'proto', '66']. [Errno 12] Cannot allocate memory.
Nov 23 19:07:52 globo-de google-accounts: ERROR Exception calling the response handler. [Errno 12] Cannot allocate memory.#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/metadata_watcher.py", line 196, in WatchMetadata#012    handler(response)#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/accounts_daemon.py", line 259, in HandleAccounts#012    self.oslogin.UpdateOsLogin(enable=False)#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/oslogin_utils.py", line 79, in UpdateOsLogin#012    status = self._GetStatus()#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/oslogin_utils.py", line 60, in _GetStatus#012    retcode = self._RunOsLoginControl('status')#012  File "/usr/lib/python2.7/dist-packages/google_compute_engine/accounts/oslogin_utils.py", line 47, in _RunOsLoginControl#012    return subprocess.call([constants.OSLOGIN_CONTROL_SCRIPT, action])#012  File "/usr/lib/python2.7/subprocess.py", line 168, in call#012    return Popen(*popenargs, **kwargs).wait()#012  File "/usr/lib/python2.7/subprocess.py", line 390, in __init__#012    errread, errwrite)#012  File "/usr/lib/python2.7/subprocess.py", line 916, in _execute_child#012    self.pid = os.fork()#012OSError: [Errno 12] Cannot allocate memory
Nov 23 19:09:37 globo-de google-ip-forwarding: WARNING Exception running ['ip', 'route', 'ls', 'table', 'local', 'type', 'local', 'scope', 'host', 'dev', 'eth0', 'proto', '66']. [Errno 12] Cannot allocate memory.
Nov 23 19:11:32 globo-de systemd[1]: phpsessionclean.service: Failed to fork: Cannot allocate memory
Nov 23 19:11:56 globo-de systemd[1]: phpsessionclean.service: Failed to run 'start' task: Cannot allocate memory
Nov 23 19:12:24 globo-de systemd[1]: Failed to start Clean php session files.
Nov 23 19:12:50 globo-de systemd[1]: phpsessionclean.service: Unit entered failed state.
Nov 23 19:13:37 globo-de systemd[1]: phpsessionclean.service: Failed with result 'resources'.
Nov 23 19:14:58 globo-de systemd[1]: apt-daily.service: Failed to fork: Cannot allocate memory
Nov 23 19:16:13 globo-de systemd[1]: apt-daily.service: Failed to run 'start' task: Cannot allocate memory

What can I do to prevent that from happening again?

Upvotes: 3

Views: 4350

Answers (2)

Fabricio Figueiredo
Fabricio Figueiredo

Reputation: 1

Another solution is to avoid swapping setting vm.swappiness to 0.

# sysctl -w vm.swappiness=0

However, it sounds me more like an OS image issue than google clouds.

Upvotes: 0

treffner
treffner

Reputation: 121

I guess the problem was that the google-cloud-compute engine vm had no swap file. So I created one with the following commands I found on google.

sudo dd if=/dev/zero of=/var/swap bs=2048 count=524288
sudo chmod 600 /var/swap
sudo mkswap /var/swap
sudo swapon /var/swap

Then I added the following line to /etc/fstab to make it permanent.

/var/swap none swap sw 0 0

Upvotes: 7

Related Questions