During NSX upgrade from 4.2.2 or 4.2.3 to 9.0.0 or 9.0.1, installing a new transport node with a lower version while the upgrade is still in progress can result in failed installation

Products

• VMware NSX

Issue/Introduction

• During NSX upgrade from 4.2.2/4.2.3 to 9.0.0/9.0.1, if a transport node with a lower version is installed while the upgrade is in progress, user configurations can not be realized on the host and host installation might fail.
• May see the following log line repeating in /var/log/cloudnet/nsx-ccp.log or /var/log/syslog.log in the NSX Manager.
  2025-10-10T11:25:51.916Z INFO NSX 86251 [nsx@4413 comp="nsx-controller" level="INFO" logger="StreamingAdapter" subcomp="nestdb-pigeon" threadName="nsx-rpc:CCP-69557ac5-####-####-####-861026ca093c:user-executor-1"] b02bd6ad-####-####-####-038f57353000: Call status for id 3 is Status(code=INVALID_ARGUMENT, msg=Unrecognized object type 1: 242\n). Disconnecting...

Environment

VMware NSX upgrading from 4.2.2 or 4.2.3 to 9.0.0 or 9.0.1

Cause

NSX versions 9.0.0 and 9.0.1 can not establish connections with NestDb server on lower version hosts successfully if the NestDb server contains empty data, which is typically the case for newly installed TransportNode. 

Resolution

This issue is fixed in NSX versions 9.0.2 and 9.1.0 to handle the Controller-host resync failure scenario.

Workaround to recover:

• Option 1: Uninstall the failed transport node and complete the remaining upgrade steps.
• Option 2: Add dummy ContiainerMsg in NestDb server.
  1. SSH as root login to ESX host.
  2. Run the following command. 
     /opt/vmware/nsx-nestdb/bin/nestdb-cli --json --beautify
  3. Enter the following line at the (nestdb) prompt. 
     put vmware.nsx.nestdb.ContainerMsg {"id" : "00000000-0000-0000-0000-000000000000"}
  4. Verify on NSX UI the transport node realization status is success
     
     Limitation: If the ESX host is rebooted before the host is upgraded, the manual workaround (steps 1-4) will need to be re-applied again.