CODEWITHSUNDEEP
javadockergoogle-compute-enginejmxvisualvm
Stevey
Stevey

Reputation: 3281

Attaching VisualVM or JConsole to Java in Docker on GCE

I am trying to get some CPU sampling working on a remote Java-in-Docker process.

I've already looked at the related questions here, and tried everything, to no avail, so I'm posting my setup here.

I have a Java process (openjdk-8) running in a Docker container on a Google Compute Engine (GCE) instance. The GCE instance and container are both running Debian-9. I want to attach VisualVM or JConsole to my Java process.

I am able to run my docker container locally and connect with both visualvm and jconsole using localhost:9010.

I start the container in the VM startup script with:

docker run -d -p 9010:9010 <my container>

The Dockerfile also has:

EXPOSE 9010

The Java process, started by the Dockerfile CMD, has the following relevant args:

"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \

I have opened up port 9010 in my gcloud firewall using:

gcloud compute firewall-rules create jmx-port --allow=tcp:9010,udp:9010

I have verified with netcat that the port is open and I can make a TCP connection to it.

I have other ports open from the same Docker container, with clients connecting successfully to those ports. They were exposed and mapped to the host ports the same way (-p port:port) and opened in the firewall the same way.

I am passing the external IP address of the GCE instance. For instance, if I do:

gcloud compute instances list

and it tells me:

NAME            ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP      STATUS
my-server-b23j  us-central1-d  n1-standard-1               10.240.0.2   108.357.213.99   RUNNING

Then I will use the argument:

108.357.213.99:9010

as the remote jmx connection host:port pair.

VisualVM and JConsole both tell me they can't connect to the remote JMX service. In both cases, I decline the secure connection, and then they say:

Cannot connect to 108.357.213.99:9010 using
service:jmx:rmi:////jndi/rmi://108.357.213.99:9010/jmxrmi

In desperation, I added a firewall rule that enables TCP/UDP connections on all ports 0-65535, but it did not make a difference -- they still could not connect.

I've read that JMX-RMI opens up anonymous ports, and that you can (at least partly?) disable this behavior by specifying both:

"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \

However, it doesn't do the trick in my case.

I've read here that you need to specify the rmi server hostname:

-Djava.rmi.server.hostname='192.168.99.100'

but my server IP is ephemeral -- it is assigned by Google Compute Engine when I create the instance, and so I can't hardwire it into the Dockerfile with the rest of the Java args.

Am I going to have to get a static IP address to make this work?

Upvotes: 4

Views: 4006

Answers (1)

Abacus
Abacus

Reputation: 19471

One possible solution would be to ssh into your GCE box and port-forward port 9010. This can be done from the local console with:

gcloud compute ssh name-of-your-gce-engine -- -L 9010:localhost:9010

Then in jconsole or jvisualvm you connect to localhost:9010. Using localhost here means that jconsole/jvisualvm will connect to your local machine, this connect is tunneled by ssh into your GCE engine and there to the host and port defined in the -L argument, which is localhost:9010, but from the GCE-engine's view. Meaning that you will end up at your application.

You still have to set the rmi server name before starting your program, but you must use

-Djava.rmi.server.hostname='localhost'

so that RMI will tell jconsole/jvisualvm to use localhost and this will then resolve to your local tunneled endpoint. And of course you still need these:

"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \

Upvotes: 6

Related Questions