SDDC Manager 9.1 upgrade fails immediately during pre-check due to degraded NSX Transport Node status

Products

• VCF Operations

Issue/Introduction

When attempting to upgrade SDDC Manager to version 9.1, the upgrade workflow fails or cancels immediately within seconds of execution.

An error similar to the following is logged in /var/log/vmware/vcf/lcm/lcm-debug.log:

Log Snippet :

DEBUG [vcf_lcm,6a069697dYYYY9f98533116b98a,e1e9,auditId=#####2b-###4-####-###3-##########5,resourceType=NSX_T_MANAGER,resourceId=fqdn] [c.v.e.s.l.p.impl.nsxt.NsxtAuditImpl,Scheduled-7] NSX Host Cluster wkld-cl01 Status is DOWN
ERROR [vcf_lcm,6a069697dYYYY9f98533116b98a,e1e9,auditId=312fce2b-deb4-YYYY-b173-144c3a035985,resourceType=NSX_T_MANAGER,resourceId=fqdn,name=fqdn] [c.v.e.s.l.p.impl.nsxt.NsxtAuditImpl,Scheduled-7] NSX Manager Transport Node : Status is DEGRADED
 DEBUG [vcf_lcm,6a069697dYYYY9f98533116b98a,e1e9,auditId=#####2b-###4-####-###3-##########5,resourceType=NSX_T_MANAGER,resourceId=fqdn,name=fqdn] [c.v.e.s.l.p.impl.nsxt.NsxtAuditImpl,Scheduled-7] NSX Edge Cluster workload EC Status is UP
ERROR [vcf_lcm,6a069697dYYYY9f98533116b98a,e1e9,auditId=#####2b-###4-####-###3-##########5,resourceType=NSX_T_MANAGER,resourceId=fqdn,name=fqdn] [c.v.e.s.l.p.impl.nsxt.NsxtAuditImpl,Scheduled-7] Setting Audit Error with status SUCCESS and compiled error message {"errorCode":"SUCCESS","errorDetails":"NSX Manager Transport Node :
Status is DEGRADED\t"}

Environment

VMware SDDC Manager 9.1

Cause

Upgrade is blocked because one or more NSX Transport Nodes (ESXi hosts) within a workload cluster are reporting a DEGRADED or DOWN status. SDDC Manager requires all fabric and transport node endpoints to be completely healthy before proceeding with a major orchestration upgrade.

Resolution

To resolve this issue, the health of the affected NSX Transport Node must be restored before retrying the upgrade:

1. Log in to the NSX Manager UI.

2. Navigate to System > Fabric > Nodes > Host Transport Nodes.

3. Identify the degraded host  and review the specific alarm details (e.g., TEP connectivity issues, configuration state sync failures, or power state).

4. Resolve the underlying host/network infrastructure issue to bring the Transport Node status back to UP / Healthy.

5. Log back into the SDDC Manager UI, re-run the upgrade pre-checks, and proceed with the SDDC Manager 9.1 upgrade once all validations pass.

 

Additional Information

If the Transport Node is degraded specifically due to a Tunnel Endpoint (TEP) IP address conflict (such as a conflict involving a PfSense Gateway or physical switch setup), please refer to the following specific Broadcom KB article:NSX Transport Node Degraded due to TEP IP Conflict with PfSense Gateway (KB 442334)