NSX-T ESXi install stuck at 0%

Products

VMware NSX

Issue/Introduction

Symptoms:

NSX-T ESXi install stuck at 0%
Customer has CA signed Certificates at hosts , which has \r\n within the certificate.
To check if the ESXi hosts have \r\n in the certificate, use below API:

GET https://<mgr-ip>/api/v1/fabric/discovered-nodes

This API will list down all discovered nodes, check the certificate of the problematic discovered node

Logs show this:

2023-03-01 15:26:39,476 ERROR [LCM] [lcm-node-0] c.v.l.c.b.BatchUpgrade [BatchUpgrade.java:1280] Attempt 2/3 of vSAN access exception. but failed with error: com.vce.lcm.exception.LCMException: Upgrade cannot continue because the NSX VIB is an incompatible version on host

2023-03-02T12:10:19.773Z ERROR task-executor-11 VcHostIssueResolveHandler 4243 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP26044" level="ERROR" subcomp="manager"] Repair failed. java.lang.IllegalArgumentException: Illegal base64 character d

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

NSX-T is unable to accept hosts certificates which has \r\n within the certificate.

Resolution

This is a known issue affecting NSX-T 3.2.2 and above and is planned to be fixed in future release.

Workaround:
Workaround is to remove \r from the certificate on the affected ESXi host.

Steps:

1. Remove \r character from the host ESXi certificate

(A). You can use any opensource tool like dos2unix to change the cert format and then replace the host certificate present at location /etc/vmware/ssl/rui.crt and then restart the services.

OR you can perform the below steps

(B). Login ssh to each ESXi that we need to fix;

    1. cat -v /etc/vmware/ssl/rui.crt // check if certificate has \r\n, \r\n is displayed as ^M

    2. cp /etc/vmware/ssl/rui.crt /etc/vmware/ssl/rui_old.crt // Take Backup

    3. sed 's/\r$//' /etc/vmware/ssl/rui.crt > /etc/vmware/ssl/rui_new.crt // Remove CR character

    4. rm -f /etc/vmware/ssl/rui.crt

    5. mv /etc/vmware/ssl/rui_new.crt /etc/vmware/ssl/rui.crt

    6. cat -v /etc/vmware/ssl/rui.crt // verify \r is removed

    7. services.sh restart

2. Wait for few minutes and verify that \r is removed. You can use below API to verify if certificate is updated correctly.
   GET https://<mgr-ip>/api/v1/fabric/discovered-nodes

3. Remove NSX from the cluster where hosts are stuck at 0%

4. Configure NSX again

Additional Information

Hosts which are already prepared prior to upgrade to 3.2.2 , with \r\n in certificates don’t have any issues.

Impact/Risks:
Unable to add / prepare host for NSX-T