During V2T migration, ESXi hosts fail to prepare with error "specified thumbprint does not match host thumbprint", or "failed to create fabric node".

Products

• VMware NSX

Issue/Introduction

Symptoms:
* NSX-T Data Center version 3.2.2.
* This issue occurs during V2T migration, during host preparation.
* Hosts will fail to prepare with one of the following errors:
   "SSL validation failed for host <host_IP> because specified thumbprint does not match host thumbprint <thumbprint>"
   "Fabric node creation failed for host <host_UUID>:<host-MOID>"
* NSX-V hosts will fail to prepare for NSX-T, with impact to the datapath on the workloads running on these hosts.
* Below logging can be observed in NSX Manager's logs:
   /var/log/migration-coordinator/migration-coordinator.log:
   2023-03-21T16:30:28.743Z csdracknsxmgrA.vcf1.csd.vblock NSX 10139 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP31807" level="ERROR" subcomp="migration-coordinator"] SSL validation failed for host 10.11.12.13 because specified thumbprint does not match host thumbprint 5da[...]79
* After reboot of the host, thumbprint error may clear any the error below can be observed:
   /var/log/migration-coordinator/migration-coordinator.log:
   2023-03-21T19:07:48.981Z INFO http-nio-127.0.0.1-7450-exec-3 MigrationFacadeImpl 10139 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="migration-coordinator"] Unit: 8a4d9bfc-xxxx-xxxx-xxxx-b11449b469bc:host-10, status: NOT_STARTED, % complete: 0.0, errors: Fabric node creation failed for host 8a4d9bfc-xxxx-xxxx-xxxx-b11449b469bc:host-10
   /var/log/syslog:
   2023-03-21T20:50:26.307Z csdracknsxmgrA.vcf1.csd.vblock NSX 1708 MONITORING [nsx@6876 comp="nsx-manager" level="WARNING" reqId="8599c4b1-2b46-4eaa-8469-cf0fe6376013" subcomp="manager" username="UC"] Entity status update not found for the provided TN ID left: 17[...]4#012right: 14[...]2#012 in table HostNodeStatusVertical_NodeStatusProperties
   2023-03-01 15:26:39,476 ERROR [LCM] [lcm-node-0] c.v.l.c.b.BatchUpgrade [BatchUpgrade.java:1280] Attempt 2/3 of vSAN access exception. but failed with error: com.vce.lcm.exception.LCMException: Upgrade cannot continue because the NSX VIB is an incompatible version on host

Environment

VMware NSX-T Data Center

Cause

NSX-T is unable to accept hosts certificates which have "\r\n" within the certificate.

Resolution

This is a known issue impacting NSX-T Data Center 3.2.2, which will be fixed in future releases.

Workaround:
Use a workaround in KB article NSX-T ESXi install stuck at 0% to remove "\r" character from the ESXi host's certificate, or renew ESXi host's certificates and initiate V2T migration.

Additional Information

Impact/Risks:
NSX-V ESXi hosts may fail to prepare for NSX-T. Datapath (connectivity) of all workloads on these hosts will be impacted.