502 Proxy Error from OpenShift DIY project

Question

On my Openshift account I have setup Tomcat 8 and JDK 8 on a DIY application with the MySql and PHPAdmin cartridges installed.

My war file points to everything correctly and there are no errors on startup in any of the logs. However, when I try to go to my OpenShift URL I receive this 502 Proxy Error in the browser. I'm using Chrome.

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

What could be causing this problem?

Answer 1

@Graham Where's the fun in that? So I'm going to share my experience, in case anybody else gets here. I think in my instance I was hitting the upper limit of authorized CPU / memory usage for my 'free' gear. Nothing really jumps out and yells "You hit the limit" but It was pretty clear something was wrong. I'm pretty happy with the results, glad I stuck it out. I've learned a whole lot about deployment to an online server with meager $$ resources.

General troubleshooting instructions start here.

First, I shut down the server hard with a $rhc app-force-stop <app_name> After that I was able to start up the system again and it would work fine. In my case I was trying to do too much with the size of server I was paying for (free!) The free server includes 512Mb Ram and 1 Gig storage. I was trying to run Node, a MongoDB and a Cron cartridge in there. Additionally I had a whole lot of asynchronous Input/Output with quite a large stack built up. In hind sight, not clever.

Error detection wasn't real easy. I didn't learn anything at all from the log files. Generally when something went wrong they just stopped recording anything at all.

There are 11 tests to do. First login to the server via SSH, and your command line tool. Note, there is no magic "you screwed up here message" You've got to look at your usage, and compare it to your authorized usage levels. So yeah, this took me awhile, but I documented this for my own notes. Here's a good place to share with others. I've learned a whole lot with this exercise. Good luck. (oh and in my case, I deleted the cron cartridge and the mongodb cartridge. I'm hosting the DB at mlab.com where its accessible from my other projects. Success for me .)

1) Memory Fail Counts: (results should be zero...)

 oo-cgroup-read memory.failcnt             // my results --> 160031
 oo-cgroup-read memory.memsw.failcnt       // my resluts --> 8572

2) Check disk Quotas

[xyz-abc.rhcloud.com 5xxx3]\> quota -s

Disk quotas for user 5xxx3 (uid 3488):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/EBSStore01-user_home01
                   608M       0   1024M           12664       0   80000

3) Check for your actual disk usage. (du = Disk Usage Sum of directories (-s) in human-readable format (-h : Byte, Kilobyte, Megabyte, Gigabyte, Terabyte and Petabyte): )

du -sh ~

du: cannot read directory `/var/lib/openshift/5xxx3/.tmp': Permission denied
du: cannot read directory `/var/lib/openshift/5xxx3/.sandbox': Permission denied
du: cannot read directory `/var/lib/openshift/5xxx3/.ssh': Permission denied
du: cannot read directory `/var/lib/openshift/5xxx3/.gearstats': Permission denied
607M    /var/lib/openshift/5xxx3/

4) List open files (lsof is a command meaning "list open files", which is used in many Unix-like systems to report a list of all open files and the processes that opened them. -n Do not resolve hostnames (no DNS). -P Do not resolve port names (list port number instead of its name). )

lsof -n -P

COMMAND    PID     USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
mongod  471639     3488   11u  IPv4 423798423      0t0  TCP 127.x.y.z:27017 (LISTEN)
node    475151     3488   10u  IPv4 423815802      0t0  TCP 127.x.y.z:8080 (LISTEN)

5) Display top CPU intensive processes (top Provide information (frequently refreshed) about the most CPU-intensive processes currently running. You do not need to include a - before options. -b Run in batch mode; don't accept command-line input. Useful for sending output to another command or to a file. -n num Update display num times, then exit.)

top -b -n 1

top - 00:48:37 up 13 days, 23:52,  0 users,  load average: 2.91, 2.27, 2.09
Tasks:  13 total,   1 running,  12 sleeping,   0 stopped,   0 zombie
Cpu(s): 11.6%us, 10.0%sy,  0.1%ni, 77.5%id,  0.5%wa,  0.0%hi,  0.2%si,  0.1%st
Mem:  15297608k total, 14537912k used,   759696k free,    36456k buffers
Swap: 52428792k total, 16372136k used, 36056656k free,  2720680k cached
   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 60898 3488      20   0 12800  968  744 R  1.9  0.0   0:00.02 top
 55776 3488      20   0  106m 2740  808 S  0.0  0.0   0:00.00 sshd
 55779 3488      20   0  104m 2260 1432 S  0.0  0.0   0:00.09 bash
432471 3488      20   0  106m  888  884 S  0.0  0.0   0:00.00 sshd
432475 3488      20   0 55144 1540 1536 S  0.0  0.0   0:00.11 sftp-server
471611 3488      20   0  9508  412  404 S  0.0  0.0   0:00.00 control
471612 3488      20   0  181m 2152 1720 S  0.0  0.0   0:00.01 logshifter
471624 3488      20   0  4072  456  448 S  0.0  0.0   0:00.00 scl
471625 3488      20   0  9236  812  808 S  0.0  0.0   0:00.00 bash
471639 3488      20   0  373m  14m  13m S  0.0  0.1   0:03.53 mongod
475123 3488      20   0  778m 5264 5172 S  0.0  0.0   0:00.08 node
475124 3488      20   0  117m 2148 1708 S  0.0  0.0   0:00.00 logshifter
475151 3488      20   0  863m 114m 6776 S  0.0  0.8   0:04.10 node

6) Review memory usage. (free -- Display statistics about memory usage: total free, used, physical, swap, shared, and buffers used by the kernel. Options: -b Calculate memory in bytes. -k Default. Calculate memory in kilobytes. -m Calculate memory in megabytes.)

free

             total       used       free     shared    buffers     cached
Mem:      15297608   14767896     529712     766468      36484    2746820
-/+ buffers/cache:   11984592    3313016
Swap:     52428792   16334312   36094480

This is where I've gone astray. There is still a tiny bit of free space, but it doesn't take me much to figure out when I'm doing an intensive I/O that I'm going to go south fast here. When that happened I didn't see any error log / messages at all. Things just stop working.

7) Check your sockets. (ss - socket statistics. The output will contain all tcp, udp and unix socket connection details. )

ss

State   Recv-Q Send-Q    Local Address:Port   Peer Address:Port

(in this case there are no open sockets.. the line above is just the column headers..)

8) Check VMstat. (vmstat – Summary information of Memory, Processes, Paging etc. Free – Amount of free/idle memory spaces. si – Swapped in every second from disk in Kilo Bytes. so – Swapped out every second to disk in Kilo Bytes. )

vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 16248996 425248  33476 2946912   88   90   321   247    4    3 12 10 78  0  0

9) Check I/O stats. (iostat – Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.)

iostat

Linux 2.6.32-573.12.1.el6.x86_64 (ex-std-node842.prod.rhcloud.com)      03/14/2016      _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.60    0.12   10.21    0.49    0.06   77.52

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
xvda              6.56       197.23       147.83  238703267  178916836
xvdf             15.08       337.29       347.44  408209376  420504392
xvdg             15.13       337.45       347.44  408413143  420502512
xvdp             65.18      1603.17      1060.59 1940282568 1283607613
dm-0              7.97       108.87        33.25  131768290   40238544
dm-1             70.00      1574.18      1060.36 1905191416 1283329611
dm-2              3.48        87.89       114.58  106366791  138678084

10) (mpstat - Report processors related statistics. )

mpstat

Linux 2.6.32-573.12.1.el6.x86_64 (ex-std-node842.prod.rhcloud.com)      03/14/2016      _x86_64_        (4 CPU)

01:10:59 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:10:59 AM  all   11.60    0.12   10.01    0.49    0.00    0.21    0.06    0.00   77.52

11) User Limits (ulimit User limits - limit the use of system-wide resources. -a All current limits are reported. )

ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 59663
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 350
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited