VCF 9.1 upgrade fails during NSX upgrade with “Invalid upgrade spec, nsxtId passed as null or blank”
search cancel

VCF 9.1 upgrade fails during NSX upgrade with “Invalid upgrade spec, nsxtId passed as null or blank”

book

Article ID: 448068

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

An upgrade to VMware Cloud Foundation 9.1 may fail during the NSX component upgrade with an error indicating that the NSX identifier in the upgrade specification is empty.

The error may occur when an incompletely deployed or unhealthy NSX Edge Transport Node remains in the NSX inventory.

Environment

VMware Cloud Foundation 9.x
VMware NSX

 

Cause

An NSX Edge Transport Node deployment did not complete successfully and remained in an unhealthy or partially registered state.

The stale Edge object prevented the NSX upgrade precheck or upgrade-specification workflow from correctly resolving the NSX inventory information. The resulting VCF upgrade specification contained a null or blank nsxtId, causing validation to fail.

The nsxtId passed as null message is a workflow validation symptom. The underlying environmental condition is the unhealthy Edge Transport Node or Edge cluster remaining in NSX inventory.

The VCF Lifecycle Management logs report an error similar to:

  • UPGRADE_SPEC_NSXT_ID_EMPTY
    Invalid upgrade spec, nsxtId passed as null, cannot be null or blank

NSX Manager logs may show an Edge Transport Node that has not completed registration:

  • Edge Tn install state: REGISTRATION_PENDING, intent version: 0

The node may later transition to a timed-out state:

  • errorCode="MP16057"
    The deployment of edge vm:
    /infra/sites/default/enforcement-points/default/
    edge-transport-nodes/<edge-node-name>
    not successful, Edge install state: REGISTRATION_TIMEDOUT

 

Log Locations

Review VCF Lifecycle Management logs on SDDC Manager:

/var/log/vmware/vcf/lcm/

Relevant files may include:

lcm-debug.log
lcm.log

Review NSX Manager API and realization logs:

/var/log/proton/nsxapi.log
/var/log/proton/nsxapi.1.log
/var/log/proton/nsxapi.2.log.gz
/var/log/proton/nsxapi.3.log.gz

Example search:

zgrep -nEi
'UPGRADE_SPEC_NSXT_ID_EMPTY|nsxtId|REGISTRATION_PENDING|REGISTRATION_TIMEDOUT|MP16057|EdgeTNRealizationUtils'
/var/log/proton/nsxapi*.log*

Resolution

Before removing any NSX Edge components, confirm that they are not providing active routing, NAT, VPN, load-balancing, DHCP, bridging, Tier-0, Tier-1, or other network services.

  1. Log in to NSX Manager.
  2. Navigate to: System > Fabric > Nodes > Edge Transport Nodes
  3. Identify Edge nodes in an unhealthy state, including nodes reporting conditions such as:
    • REGISTRATION_PENDING
    • REGISTRATION_TIMEDOUT
    • Deployment Failed
    • Disconnected
    • Down
  4. Review the affected Edge node and determine whether it belongs to an Edge cluster.
  5. Verify that the Edge node and Edge cluster are not referenced by any active networking configuration.
  6. If the affected Edge node and Edge cluster are unused, remove the failed Edge node from NSX.
  7. Remove the associated Edge cluster only when it is also unused and contains no required Edge nodes.
  8. Confirm that the remaining NSX components are healthy.
  9. Return to the VCF Lifecycle Management interface and retry or restart the failed upgrade workflow.
Powered by Wolken Software