NSX config Realization shows failed or DOWN.

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX Intelligence may be stuck in degraded state or deployment itself may fail.
• Logical Switch Realization fails but works after all 3x UAs restart.
• Host may show unknown status, manager GUI may show controller connectivity unavailable.
  
  

Environment

VMware NSX-T Data Center

Cause

• If the NSX Manager certificate contains a carriage return character, the deployment of the NSX Intelligence appliance might fail or the NSX Intelligence appliance might get stuck in a degraded state after it has been deployed.
• If the NSX Manager certificate or NSX Intelligence certificate that has a carriage return character is updated after the NSX Intelligence appliance has been deployed, and when multiple manager services are restarted simultaneously, some of the services might fail to initialize.

Note : Issue is only seen when there is Intelligence installed in the environment with s certificate that includes carriage return.

Resolution

Logs to look for and their location
----------------------------------

++ /var/log/syslog

$ grep "unable to create new native thread" syslog* | wc -l
386

++ /var/log/nsxapi.log
$ grep "java.lang.OutOfMemoryError: unable to create new native thread" nsxapi.* | wc -l
71766

++ desired_state_manager.json 

Search for MGMT_CLUSTER and check if there is "\n" carriage return at the end of certificate as shown below.



        "pem_encoded": "-----BEGIN CERTIFICATE-----\nMIIETzCCAzegAwIBAgIJAMZD53PGKIXqMA0GCSqGSIb3DQEBCwUAMIGhMQswCQYDVQQDDAJDQTEX\nMBUGCgmSJo
mT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMC\nVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExIzAhBgNVBAoMGmV1bmwxbWd2Y3NhMDEuY2FybGlzb
GUu\nY29tMRswGQYDVQQLDBJWTXdhcmUgRW5naW5lZXJpbmcwHhcNMjAwOTIyMjAyOTQ4WhcNMzAwODAx:
:
:
:
zoUFkSLuRBqVeL7ndA1lOTGyT0eR
HRitv0bBLKpad9fp4v\nL8Y+9/IVVZc7/IJG2faagoD6IcEbDPJmicxEuxVfrpTtyAv9b3lliAPlgL2lEOK22KjM7vswI2nB\nxpeK9EVZ24B34cL568iv/x1t3ywLOqemTBwVQzYDV
fcaWVhM3YiuVi1whl777A0=\n-----END CERTIFICATE-----\n",
        "resource_type": "certificate_signed",
        "tags": [],
        "used_by": [
          {
            "node_id": "92871242-####-####-####-########ba9",
            "service_types": [
              "MGMT_CLUSTER"

Issue is resolved in NSX-T 3.1.1 shipped with NSX Intelligence 1.2.0 and later releases of NSX-T/NSX-I.

Workaround:
Use one of the following to work around the issue:

    1. Use a certificate that does not contain a carriage return character.

             or
    2. Check to see if manager service is in a Down or degraded state on any MP node and restart it

        restart service manager

Additional Information

Impact/Risks:

• New config realization fails
• DP impact for some VMs. vMotion of VMs to different host brings back the connectivity.