DNAT Rules Failing to Sync Due to Expired Transport Node Certificate - VMware NSX
search cancel

DNAT Rules Failing to Sync Due to Expired Transport Node Certificate - VMware NSX

book

Article ID: 453107

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Newly published DNAT or firewall rules fail to sync to specific Edge or Host Transport Nodes. Inbound connections from external clients to the public NAT address fail to establish.

Symptoms

  • NSX Manager reports one or more ESXi hosts or Edge nodes as Host Disconnected.
  • Management connectivity states show Manager Connectivity Down or Controller Connectivity Unknown.
  • Heartbeat failures are observed between the NSX Manager and the Transport Node.
  • Log analysis on the Transport Node reveals communication errors such as: error: 335544539-short read.

Environment

VMware NSX

Cause

The Transport Node certificate has expired, preventing the Management Plane Agent (MPA) from establishing a secure communication channel with the NSX Manager. This failure prevents the distribution of configuration updates, including new DNAT rules or firewall policies.

Resolution

The resolution for this article is as per the article 345825

Manual intervention is required for any node originally deployed on 4.1.x/4.2.0.

For NSX versions from 4.1.0 through to 4.2.0 inclusive:

  • Check the connection status of the Transport Node on the NSX UI, System -> Fabric -> Hosts/Nodes

Note:

  • A Host or Edge may still show as Connected and Success even though the Certificate is expired.
  • Validate the expiry date of the certificate as root on the Host or Edge, using the command:
    openssl x509 -enddate -noout -in /etc/vmware/nsx/host-cert.pem.

Transport Node has an expired or expiring certificate but is still connected to NSX:

Transport Node certificate has expired and TN is in a disconnected state in NSX:

  • The CARR script can not be used to replace the TN certificates in this scenario.
    1. SSH to the Transport Node as root user.
    2. Empty Transport Node certificate and private key:

      cat /dev/null > /etc/vmware/nsx/host-cert.pem
      cat /dev/null > /etc/vmware/nsx/host-privkey.pem

    3. Generate a new self-signed TN certificate and key:

      For NSX 4.1.2.5 and higher, restarting the nsx-proxy service creates the new cert-key pair:(Move to Step 4 post this):
      /etc/init.d/nsx-proxy restart

      For NSX 4.1.x versions prior to 4.1.2.5:

      1. Create a temporary OpenSSL config file from the existing OpenSSL config:
        • cat /etc/vmware/nsx/openssl-proxy.cnf > /tmp/tmp-openssl-proxy.cnf 
      2. UUID is extracted and added to the temporary OpenSSL config.
        • echo "UID = $(grep -o '<uuid>[^<]*' /etc/vmware/nsx/host-cfg.xml | sed 's/<uuid>//')" >> /tmp/tmp-openssl-proxy.cnf
      3. Add extension in the temporary OpenSSL config.
        • echo -e "[ req_ext ]\nbasicConstraints     = CA:FALSE\nextendedKeyUsage     = clientAuth\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer" >> /tmp/tmp-openssl-proxy.cnf
      4. Replace the certificate, where below -days parameter specifies 3650 days (10 years) validity period.
        • openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -keyout /etc/vmware/nsx/host-privkey.pem -out /etc/vmware/nsx/host-cert.pem -config /tmp/tmp-openssl-proxy.cnf -extensions req_ext
    4. Identify NSX Manager thumbprint, SSH as admin user to NSX Manager:

      get certificate api thumbprint

    5. To push the new cert-key pair to the Manager, from root user on the Host or Edge run (Any NSX Manager name or IP can be used)
      • For the Edges (the manager ip that should be used will be the manager that is disconnected in the output of get controllers)
        • su admin -c push host-certificate <Manager hostname-or-IP> username admin thumbprint <thumbprint from step 4>
        • su admin -c sync-aph-certificates <Manager hostname-or-IP> username admin thumbprint <thumbprint from step 4>
      • For the Hosts (the manager ip that should be used will be the manager that is disconnected in the output of get controllers)
        • nsxcli -c push host-certificate <Manager hostname-or-IP> username admin thumbprint <thumbprint from step 4>
        • nsxcli -c sync-aph-certificates <Manager hostname-or-IP> username admin thumbprint <thumbprint from step 4>

                         Note: the get controllers command might show an old disconnected NSX manager. Running the above commands might result in the following error:
             % Push certificate failed: 'internal error'
                          In that case, run the get controllers command again and this time, it will show the right NSX manager IP.

              6. In the NSX UI, navigate back to the host under System-->Fabric-->Hosts and resolve the alarm by selecting the host disconnected link.

              7. If the Transport Node status still reflects an error, it is sometimes necessary to restart the nsx-proxy and nsx-opsagent on the Transport Node to restore this connection.

      • Edge:
        • /etc/init.d/nsx-proxy restart
        • /etc/init.d/nsx-opsagent-appliance restart
      •  Host:
        • /etc/init.d/nsx-proxy restart
        • /etc/init.d/nsx-opsagent restart

Additional Information

  • If this issue recurs, verify the expiration dates of the certificates regularly.
  • For defects and enhancements related to certificate management, you may subscribe to this article to be updated on fix status.
  • To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.
Powered by Wolken Software