NSX-T ESXi install stuck at 0%

Products

VMware NSX

Issue/Introduction

NSX-T ESXi install stuck at 0%
Customer has CA signed Certificates at hosts , which has \n within the certificate.
To check if the ESXi hosts have \n in the certificate, use below API:
```
GET https://<mgr-ip>/api/v1/fabric/discovered-nodes
```

This API will list down all discovered nodes, check the certificate of the problematic discovered node

Logs show this:

2023-03-01 15:26:39,476 ERROR [LCM] [lcm-node-0] c.v.l.c.b.BatchUpgrade [BatchUpgrade.java:1280] Attempt 2/3 of vSAN access exception. but failed with error: com.vce.lcm.exception.LCMException: Upgrade cannot continue because the NSX VIB is an incompatible version on host 


2023-03-02T12:10:19.773Z ERROR task-executor-11 VcHostIssueResolveHandler 4243 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP26044" level="ERROR" subcomp="manager"] Repair failed. java.lang.IllegalArgumentException: Illegal base64 character d

Environment

VMware NSX-T Data Center 3.x

VMware NSX

Cause

NSX-T is unable to accept hosts certificates which has \n within the certificate.

Resolution

This is a known issue affecting NSX-T 3.2.2 and above and is planned to be fixed in future release.

Workaround:
Remove from the certificate on the affected ESXi host.

Steps:

1. Remove character from the host ESXi certificate

(A). You can use any opensource tool like dos2unix to change the cert format and then replace the host certificate present at location /etc/vmware/ssl/rui.crt and then restart the services.

OR you can perform the below steps

(B). Login ssh to each ESXi that we need to fix;

check if certificate has \n, \n is displayed as ^M
```
cat -v /etc/vmware/ssl/rui.crt
```

Take Backup

cp /etc/vmware/ssl/rui.crt /etc/vmware/ssl/rui_old.crt

Remove CR character

sed 's/$//' /etc/vmware/ssl/rui.crt > /etc/vmware/ssl/rui_new.crt

rm -f /etc/vmware/ssl/rui.crt

mv /etc/vmware/ssl/rui_new.crt /etc/vmware/ssl/rui.crt

verify is removed
```
cat -v /etc/vmware/ssl/rui.crt
```
Restart services
```
services.sh restart
```

2. Wait for few minutes and verify that is removed. You can use below API to verify if certificate is updated correctly.

GET https://<mgr-ip>/api/v1/fabric/discovered-nodes

3. Remove NSX from the cluster where hosts are stuck at 0%

4. Configure NSX again

Additional Information

Hosts which are already prepared prior to upgrade to 3.2.2 , with \n in certificates don’t have any issues.

Impact/Risks:
Unable to add / prepare host for NSX-T