Adding Compute Manager in NSX fails with "IllegalStateException: connection not yet open"
search cancel

Adding Compute Manager in NSX fails with "IllegalStateException: connection not yet open"

book

Article ID: 319050

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  1. Adding Compute Manager in NSX UI in NSX UI fails with: 
    "Compute manager server <CM hostname> could not be connected, server might be un-reachable or connection details might be invalid. Please check if compute manager certificate is valid and not revoked. If the issue persists, please check whether the https and http ports of compute manager are open in the firewall on all NSX nodes. (Error code: 7058)"

    Example in UI:



    On the NSX Manager, /var/log/cm-inventory/cm-intentory.log shows the input and provided thumbprint match the current VC thumbprint, but there is an Exception in getting vCenter Server thumbprint with "IllegalStateException: connection not yet open".
  2. Get the VC cert thumbprint from vCenter CLI as root:
  3. root@vcenter [ ~ ]# openssl x509 -in /etc/vmware-vpx/ssl/rui.crt -fingerprint -sha256 -noout
  4. Check cm-intentory.log for the following logging:

    20xx-xx-xxTxx:xx:xx.xxxZ  INFO http-nio-127.0.0.1-7443-exec-5 VcUtilsImpl 62198 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="cm-inventory" username="admin"] input thumbprint <correct thumbprint hash>
    20xx-xx-xxTxx:xx:xx.xxxZ  INFO http-nio-127.0.0.1-7443-exec-5 VcUtilsImpl 62198 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="cm-inventory" username="admin"] provided thumbprint is <correct thumbprint hash>
    20xx-xx-xxTxx:xx:xx.xxxZ  INFO http-nio-127.0.0.1-7443-exec-5 MultiFileWatcher 62198 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="cm-inventory" username="admin"] Start watching file /config/manager/security.properties
    20xx-xx-xxTxx:xx:xx.xxxZ  INFO http-nio-127.0.0.1-7443-exec-5 NsxTrustManager 62198 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="cm-inventory" username="admin"] Check revocation
    20xx-xx-xxTxx:xx:xx.xxxZ  INFO http-nio-127.0.0.1-7443-exec-5 NsxTrustManager 62198 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="cm-inventory" username="admin"] Trust thumbprint of OU=XXX,O=XXX,L=XXX,ST=XXX,C=XX,CN=XXX
    20xx-xx-xxTxx:xx:xx.xxxZ  INFO http-nio-127.0.0.1-7443-exec-5 NsxTrustManager 62198 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="cm-inventory" username="admin"] checkServerTrusted: OU=XXX,O=XXX,L=XXX,ST=XXX,C=XX,CN=XXX for authType=ECDHE_RSA succeeded.
    20xx-xx-xxTxx:xx:xx.xxxZ  WARN http-nio-127.0.0.1-7443-exec-5 VcUtilsImpl 62198 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" reqId="<UUID>" subcomp="cm-inventory" username="admin"] Exception in getting vc thumbprint
    java.lang.IllegalStateException: connection not yet open
            at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getServerCertificates(AbstractDelegateHttpsURLConnection.java:268) ~[?:1.8.0_362]
            at sun.net.www.protocol.https.HttpsURLConnectionImpl.getServerCertificates(HttpsURLConnectionImpl.java:211) ~[?:1.8.0_362]
    .
    .
    .
    20xx-xx-xxTxx:xx:xx.xxxZ  INFO http-nio-127.0.0.1-7443-exec-5 CmPluginManagerImpl 62198 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="cm-inventory" username="admin"] Compute manager thumbprint com.vmware.nsx.management.cminv.plugin.CmThumbprint@XXXXXXX
    20xx-xx-xxTxx:xx:xx.xxxZ  WARN http-nio-127.0.0.1-7443-exec-5 CmInventoryService 62198 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" reqId="<UUID>" subcomp="cm-inventory" username="admin"] Failed to retrieve computer manager <UUID> thumbprint
    20xx-xx-xxTxx:xx:xx.xxxZ ERROR http-nio-127.0.0.1-7443-exec-5 JsonRpcErrorResolver 62198 - [nsx@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="cm-inventory"] Error discovered by JSON RPC
    com.vmware.nsx.management.cminv.common.exceptions.InventoryException: Compute manager server <IP> could not be connected, server might be un-reachable or connection details might be invalid. Please check if compute manager certificate is valid and not revoked. If the issue persists, please check whether the https and http ports of compute manager are open in the firewall on all NSX nodes.
            at com.vmware.nsx.management.cminv.computemanager.service.ComputeManagerServiceImpl.validateComputeManagerThumbprint(ComputeManagerServiceImpl.java:319) ~[libcm-inv-container.jar:?]

  5. NSX is able to successfully connect to vCenter on ports 443, 80, and a TLS connection succeeds:
    root@nsxmgr:~# nc -vz <VC IP> 443
    Connection to <VC IP> 443 port [tcp/https] succeeded!
    root@nsxmgr:~# nc -vz <VC IP> 80
    Connection to <VC IP> 80 port [tcp/http] succeeded!


    root@nsxmgr:~# openssl s_client -connect <VC IP>:443
    CONNECTED(00000003) 
    Note: It is normal to see "verify error:num=21:unable to verify the first certificate"


    4. Trying to pull the vCenter landing page via curl fails from the NSX Manager:
    root@nsxmgr:/var/log/cm-inventory# curl -k https://<VC IP> -v
    *   Trying <VC IP>:443...
    …
    …
    < HTTP/2 404


    5. The /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd.conf file on vCenter is missing a root "/" endpoint entry. For example:
    root@vcenter [ ~ ]# cat /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd.conf
    # Endpoint               Connection-type  Endpoint-address                     HTTP-access-Mode  HTTPS-access-mode
    # To support direct access to vpxd without going through API proxy
    /Query found 8080 allow reject
    /vsan/sdkTunnel namedpipetunnel /var/run/vmware-vsan/vsan-sdkTunnel-pipe allow reject
    root@vcenter [ ~ ]#

Cause

This is caused by a rare vCenter issue where the root endpoint is missing from the /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd.conf file.

Resolution

No vCenter resolution at this time. See workaround to adding missing endpoint to vpxd.conf.

Workaround:
Add the missing root endpoint to vpxd.conf.

  1. Create a backup of the vpxd.conf file on vCenter:
    root@vcenter[ ~ ]# cp /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd.conf /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd.conf.bk
  2. Add the missing root endpoint to vpxd.conf.
    root@vcenter[ /etc/vmware-rhttpproxy/endpoints.conf.d ]# vim vpxd.conf
    root@vcenter[ /etc/vmware-rhttpproxy/endpoints.conf.d ]# cat vpxd.conf
    # Endpoint               Connection-type  Endpoint-address                     HTTP-access-Mode  HTTPS-access-mode
    # To support direct access to vpxd without going through API proxy
    /Query found 8080 allow reject
    /vsan/sdkTunnel namedpipetunnel /var/run/vmware-vsan/vsan-sdkTunnel-pipe allow reject
    / namedpipe /var/run/vpxd/vpxd-webserver-pipe redirect allow
    root@vcenter[ /etc/vmware-rhttpproxy/endpoints.conf.d ]#
  3. Run the below command so that rhttpproxy reloads the endpoints:
    kill -1 `pidof rhttpproxy`

Alternatively, reboot the VCSA Virtual Machine.

Additional Information

Impact/Risks:
Compute Manager registration with NSX fails.

Powered by Wolken Software