"ssh: rejected: connect failed" error when accessing JMX in JConsole in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

We can remotely monitor Java applications deployed on Tanzu Application Service for VMs (TAS for VMs) with Java Management Extensions (JMX).

To enable Java Management Extensions (JMX) for Java applications running on TAS for VMs, refer to How to remotely monitor Java applications deployed on Tanzu Application Service for VMs with Java Management Extensions 

However, when trying to enable JMX, you hit the following error when accessing the JMX dashboard:

connect to localhost:5000 failed: ssh: rejected: connect failed (dial tcp 127.0.0.1:5000: connect: connection refused)

Environment

Product Version: 2.11

Resolution

There are several possible reasons for this "connection refused" error. One common reason is JMX not being enabled successfully.

SSH into the app container and see whether the port is in a "LISTEN" state. For example, if it is correctly configured, port 5000 is ready for new connections.

vcap@811eccc2-730f-438f-5673-eea0:~$ netstat -lntp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      15/java             
tcp        0      0 0.0.0.0:61001           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:61002           0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:61003         0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:2222            0.0.0.0:*               LISTEN      20/diego-sshd       
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      15/java             
tcp        0      0 0.0.0.0:37009           0.0.0.0:*               LISTEN      15/java             
vcap@811eccc2-730f-438f-5673-eea0

If you do not see the specific port is in a "LISTEN" state inside the container, it means the JMX is not configured properly.

You can work around this issue by adding JAVA_OPTS instead of populating JBP_CONFIG_JMX.

For example, we can append the following JAVA_OPTS to the environment variables in the "cf set-env" command line or in the manifest file. After appending JAVA_OPTS to the environment variables, run "cf restage" for this change to be successful.

  env:
    JAVA_OPTS: '-Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.s
un.management.jmxremote.port=5000 -Dcom.sun.management.jmxremote.rmi.port=5000