Unstretch of a vSAN cluster from SDDC fails at the task 'Remove AZ2 NSX uplink profiles'

Products

• VMware NSX
• VMware SDDC Manager

Issue/Introduction

• When unstretching the cluster it fails with at the task "Remove AZ2 NSX uplink profiles" with the following message:

 "Unable to delete Host Switch Uplink Profile"

• In the SDDC domainmanager.log a log similar to the following is seen which indicates failing NSX API invocation when trying to delete the UplinkProfile:

ERROR [vcf_dm,<UUID>] [c.v.v.c.n.s.c.c.ComplexHelpers,dm-exec-3]  Exception occurred during NSX API invocationjava.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.InvalidRequest: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) => {    messages = [],    data = struct => {error_message=Cannot update or delete the UplinkProfile [<UUID>] because it is used by 5 TransportNodes. Please use the object name or UUID in Global Search to find all linked objects., httpStatus=BAD_REQUEST, error_code=8504, module_name=NsxSwitching service},    errorType = INVALID_REQUESTCaused by: com.vmware.vapi.std.errors.InvalidRequest: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) => {    messages = [],    data = struct => {error_message=Cannot update or delete the UplinkProfile [<UUID>] because it is used by 5 TransportNodes. Please use the object name or UUID in Global Search to find all linked objects., httpStatus=BAD_REQUEST, error_code=8504, module_name=NsxSwitching service},    errorType = INVALID_REQUEST

• In the NSX Manager /var/log/syslog messages similar to the following will be seen:

<Manager Hostname> NSX 5194 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="manager"] UserName:'admin' ModuleName:'NsxSwitching service' Operation:'DELETE@/api/v1/infra/host-switch-profiles/<UPLINK PROFILE UUID>' Operation status: 'failure' Error: Cannot update or delete the UplinkProfile [<UPLINK PROFILE UUID>] beCause it is used by 5 TransportNodes. Please use the object name or UUID in Global Search to find all linked objects.

 

Environment

VMware SDDC Manager 5.2.1

Cause

• Known issue were a NSX host\Transport Node(s) are not properly deleted but are moved out of the cluster.
• Since the TN configuration is not deleted the TN acts as an individual TN registered from NSX.
• The TN(s) have a reference to the uplink profile, so the profile is locked from being deleted. 

Resolution

• Issue is fixed in VCF 5.2.2.

Workaround:

1. Use the search feature in the NSX UI to find all host objects associated with the uplink profile UUID referenced in the Error log message.
2. Once identified go to Fabric-> Hosts-> Standalone Hosts.
3. Select the hosts identified in step 1 and click 'Delete NSX'.
4. Rerun the decommission workflow in SDDC after it is confirmed that the hosts no longer appear in the NSX fabric.