jps returns no output even when java processes are running

Question

I'm trying to debug some issues with java processes on a Solaris box, but running jps returns no output. And jstack gives the error 'Permission denied'. The box is part of a cluster of 3 identical servers, jps and jstack work fine on the 2 other servers.

I found the following forum post from someone with the same problem but no answers: http://forums.sun.com/thread.jspa?threadID=5422237

To clarify running bps and grep for java gives all java proceses correctly, but jps gives nothing (anonymised with 'program' and 'client' to protect the guilty):

program @ clientdelivery2 : ~/
-> bps auxww|grep java
program     3427  5.5 54.067742726649544 ?        S   Sep 25 1039:47 /usr/jdk/instances/jdk1.6.0_16/bin/amd64/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/app/client/program/tomcat/conf/logging.properties -Xmx6144m -XX:PermSize=128m -XX:MaxPermSize=512m -Djava.endorsed.dirs=/app/client/program/tomcat/endorsed -classpath :/app/client/program/tomcat/bin/bootstrap.jar -Dcatalina.base=/app/client/program/tomcat -Dcatalina.home=/app/client/program/tomcat -Djava.io.tmpdir=/app/client/program/tomcat/temp org.apache.catalina.startup.Bootstrap start
program    29915  0.1 11.915252441467896 ?        S 14:55:28  3:59 /usr/jdk/instances/jdk1.6.0_16/bin/amd64/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/app/clientclone/program/tomcat/conf/logging.properties -Xmx2g -XX:PermSize=128m -XX:MaxPermSize=512m -Dcom.sun.management.jmxremote -Djava.endorsed.dirs=/app/clientclone/program/tomcat/endorsed -classpath :/app/clientclone/program/tomcat/bin/bootstrap.jar -Dcatalina.base=/app/clientclone/program/tomcat -Dcatalina.home=/app/clientclone/program/tomcat -Djava.io.tmpdir=/app/clientclone/program/tomcat/temp org.apache.catalina.startup.Bootstrap start
program     1573  0.0  0.0 4760 1332 pts/5    S 17:05:24  0:00 grep --colour java

program @ clientdelivery2 : ~/
-> jps

program @ clientdelivery2 : ~/
->

I asked around and from here http://forums.oracle.com/forums/message.jspa?messageID=5408592 I've got that the issue is:

12460/2: mkdir("/tmp/hsperfdata_program", 0755) Err#13 EACCES [ALL]

Meaning jps is being denied access to the psperfdata directory.

Has anyone run into this issue and know how to solve it?

Answer 1

Make sure the program you are trying to 'detect' with jps (and jstack, incidentally) runs without setting the java.io.tmpdir setting, or setting it to the system default.

There are a number of bugs on the Sun Developer Network temp dir locations should not be hardcoded, Fix for 6938627 breaks visualvm monitoring when -Djava.io.tmpdir and Make temporary directory use property java.io.tmpdir which are relevant here.

The story: java Java 6 Update 22 used to use a hardcoded temporary directory for putting data gathered for use by jps and jstack. The jps and jstack programs knew where to look.

However, because someone raised a 'bug' in Java 6 Update 23 they 'fixed' it to use the java.io.tmpdir java runtime setting instead. Now, this defaults to a system-specific location, which is what the 'hardcoded' one was. But if you set the option when invoking your java program, then it will use that instead. Result: jps and jstack look where they expect it to be and don't find anything.

The solution is therefore to ensure that the java.io.tmpdir option is set to the system default (e.g., on the Mac:

> java -Djava.io.tmpdir=$TMPDIR javamain

)

when invoking your program. Then jps and jstack will find it.

My colleague Glyn Normington describes this on his blog. There is apparently a fix in Java 6 Update 25.

Answer 2

Also, make sure that your startup script does not include -XX:+PerfDisableSharedMem because with this option the JVM will not write any stats, making the process invisible to jps and jstat.

See https://support.datastax.com/hc/en-us/articles/208269876-Java-utilities-such-as-jps-or-jstat-unable-to-monitor-DSE-processes for details.

Answer 3

tldr: sudo jps worked for me (for reasons invoked in other answers)

Answer 4

Are you running jps as the same user that is running the Java processes? Even if you run jps as root, it will only return the processes run by that user (root, in this case).

Answer 5

Turns out the user didn't have access to /tmp because of some issue with mounting the file system. This leads to the files inside hsperfdata_ to never be written, even though the user had access to the /tmp/hsperfdata_ folder itself.

Answer 6

Try:

jps -J-Djava.io.tmpdir=/app/client/program/tomcat/temp