NSX host cluster upgrade might be disabled on the UI for a few ESXi host Clusters on SDDC Manager

Products

• VMware SDDC Manager
• VMware vSphere ESXi 8.0

Issue/Introduction

• NSX host cluster upgrade might be disabled on the UI for a few ESXi host Clusters on SDDC Manager due to periodic cluster health checks
• However upgrade can be performed using the API
• The upgrade is failing even though the target cluster is healthy, simply because a host in a non-targeted/unrelated cluster is marked as DOWN in NSX Manager.
• The transport node status is DEGRADED, leading SDDC Manager to mark the entire NSX host cluster as DOWN, thus halting the upgrade.
• LCM Logs on SDDC confirm several hosts (including the one in the unrelated cluster) are showing DEGRADED status.

/var/log/vmware/vcf/lcm/lcm.log

lcm.202#-0#-##.2.log-debug:202#-0#-##T13:58:49.416+0000 ERROR [vcf_lcm,###################15d4,1afb,auditId=####-####-####-####-########61a8,resourceType=NSX_T_MANAGER,resourceId=esxi.domain.com,name=
esxi.doamin.com] [c.v.e.s.l.p.i.n.NsxtInventoryLoader,vac-scheduler-1] Setting NSX host cluster status : DOWN, since transport node status is DEGRADED for transport-node.domain.com
lcm.202#-0#-##.2.log-debug:202#-0#-##T13:58:49.416+0000 ERROR [vcf_lcm,###################15d4,1afb,auditId=###-####-####-####-########61a8,resourceType=NSX_T_MANAGER,resourceId=esxi.domain.com,name=esxi.domain.com

Environment

ESXi 8.x 

SDDC 5.2.1.x

NSX 4.x

 

Cause

If a BFD (Bidirectional Forwarding Detection) tunnel is down, it's expected behavior in NSX for other cluster Transport Nodes to appear degraded due to a few tunnels being down.

Resolution

Ensure the tunnels are up before attempting NSX host cluster upgrade.

Please reach out to Broadcom Support to validate the environment and take corrective actions accordingly.

Note: Service(s) restart on SDDC manager is required if any changes made to the SDDC config file(s) during the troubleshooting.