NSX config Realization shows failed or DOWN.
Article ID: 327290
Updated On:
Products
VMware NSX Networking
Issue/Introduction
Symptoms:
- NSX Intelligence may be stuck in degraded state or deployment itself may fail.
- Logical Switch Realization fails but works after all 3x UAs restart.
- Host may show unknown status, manager GUI may show controller connectivity unavailable.
- NSX Intelligence may be stuck in degraded state or deployment itself may fail.
- Logical Switch Realization fails but works after all 3x UAs restart.
- Host may show unknown status, manager GUI may show controller connectivity unavailable.
Environment
VMware NSX-T Data Center
Cause
- If the NSX Manager certificate contains a carriage return character, the deployment of the NSX Intelligence appliance might fail or the NSX Intelligence appliance might get stuck in a degraded state after it has been deployed.
- If the NSX Manager certificate or NSX Intelligence certificate that has a carriage return character is updated after the NSX Intelligence appliance has been deployed, and when multiple manager services are restarted simultaneously, some of the services might fail to initialize.
Issue is only seen when there is Intelligence installed in the environment with certificate that includes carriage return.
- If the NSX Manager certificate or NSX Intelligence certificate that has a carriage return character is updated after the NSX Intelligence appliance has been deployed, and when multiple manager services are restarted simultaneously, some of the services might fail to initialize.
Issue is only seen when there is Intelligence installed in the environment with certificate that includes carriage return.
Resolution
Logs to look for and their location
----------------------------------
++ /var/log/syslog
$ grep "unable to create new native thread" syslog* | wc -l
386
++ /var/log/nsxapi.log
$ grep "java.lang.OutOfMemoryError: unable to create new native thread" nsxapi.* | wc -l
71766
++ desired_state_manager.json
Search for MGMT_CLUSTER and check if there is "\r\n" carriage return at the end of certificate as shown below.
"pem_encoded": "-----BEGIN CERTIFICATE-----\nMIIETzCCAzegAwIBAgIJAMZD53PGKIXqMA0GCSqGSIb3DQEBCwUAMIGhMQswCQYDVQQDDAJDQTEX\r\nMBUGCgmSJo
mT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMC\r\nVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExIzAhBgNVBAoMGmV1bmwxbWd2Y3NhMDEuY2FybGlzb
GUu\r\nY29tMRswGQYDVQQLDBJWTXdhcmUgRW5naW5lZXJpbmcwHhcNMjAwOTIyMjAyOTQ4WhcNMzAwODAx:
:
:
:
zoUFkSLuRBqVeL7ndA1lOTGyT0eR
HRitv0bBLKpad9fp4v\r\nL8Y+9/IVVZc7/IJG2faagoD6IcEbDPJmicxEuxVfrpTtyAv9b3lliAPlgL2lEOK22KjM7vswI2nB\r\nxpeK9EVZ24B34cL568iv/x1t3ywLOqemTBwVQzYDV
fcaWVhM3YiuVi1whl777A0=\r\n-----END CERTIFICATE-----\n",
"resource_type": "certificate_signed",
"tags": [],
"used_by": [
{
"node_id": "92871242-2f61-3662-e164-56e8abb1cba9",
"service_types": [
"MGMT_CLUSTER"
Issue is resolved in NSX-T 3.1.1 shipped with NSX Intelligence 1.2.0 and later releases of NSX-T/NSX-I.
Workaround:
Use one of the following information to work around the issue.
1. Use a certificate that does not contain a carriage return character.
or
2. Check to see if manager service is in a Down or degraded state on any MP node and restart it
restart service manager
Additional Information
Impact/Risks:
- New config realization fails
- DP impact for some VMs. vMotion of VMs to different host brings back the connectivity.
- New config realization fails
- DP impact for some VMs. vMotion of VMs to different host brings back the connectivity.
Feedback
Yes
No
Powered by
