NSX config Realization shows failed or DOWN.

Products

• VMware NSX

Issue/Introduction

Symptoms:
- NSX Intelligence may be stuck in degraded state or deployment itself may fail.
- Logical Switch Realization fails but works after all 3x UAs restart.
- Host may show unknown status, manager GUI may show controller connectivity unavailable.

Environment

VMware NSX-T Data Center

Cause

• If the NSX Manager certificate contains a carriage return character, the deployment of the NSX Intelligence appliance might fail or the NSX Intelligence appliance might get stuck in a degraded state after it has been deployed.
• If the NSX Manager certificate or NSX Intelligence certificate that has a carriage return character is updated after the NSX Intelligence appliance has been deployed, and when multiple manager services are restarted simultaneously, some of the services might fail to initialize.
  
  
  Issue is only seen when there is Intelligence installed in the environment with certificate that includes carriage return.

Resolution

Logs to look for and their location
----------------------------------

++ /var/log/syslog

$ grep "unable to create new native thread" syslog* | wc -l
386

++ /var/log/nsxapi.log
$ grep "java.lang.OutOfMemoryError: unable to create new native thread" nsxapi.* | wc -l
71766

++ desired_state_manager.json 

Search for MGMT_CLUSTER and check if there is "\r\n" carriage return at the end of certificate as shown below.



        "pem_encoded": "-----BEGIN CERTIFICATE-----\nMIIETzCCAzegAwIBAgIJAMZD53PGKIXqMA0GCSqGSIb3DQEBCwUAMIGhMQswCQYDVQQDDAJDQTEX\r\nMBUGCgmSJo
mT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMC\r\nVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExIzAhBgNVBAoMGmV1bmwxbWd2Y3NhMDEuY2FybGlzb
GUu\r\nY29tMRswGQYDVQQLDBJWTXdhcmUgRW5naW5lZXJpbmcwHhcNMjAwOTIyMjAyOTQ4WhcNMzAwODAx:
:
:
:
zoUFkSLuRBqVeL7ndA1lOTGyT0eR
HRitv0bBLKpad9fp4v\r\nL8Y+9/IVVZc7/IJG2faagoD6IcEbDPJmicxEuxVfrpTtyAv9b3lliAPlgL2lEOK22KjM7vswI2nB\r\nxpeK9EVZ24B34cL568iv/x1t3ywLOqemTBwVQzYDV
fcaWVhM3YiuVi1whl777A0=\r\n-----END CERTIFICATE-----\n",
        "resource_type": "certificate_signed",
        "tags": [],
        "used_by": [
          {
            "node_id": "92871242-####-####-####-########ba9",
            "service_types": [
              "MGMT_CLUSTER"

Issue is resolved in NSX-T 3.1.1 shipped with NSX Intelligence 1.2.0 and later releases of NSX-T/NSX-I.

Workaround:
Use one of the following to work around the issue:

    1. Use a certificate that does not contain a carriage return character.

             or
    2. Check to see if manager service is in a Down or degraded state on any MP node and restart it

        restart service manager

Additional Information

Impact/Risks:
- New config realization fails
- DP impact for some VMs. vMotion of VMs to different host brings back the connectivity.