Removing a host in VCF on VxRail fails at “Get VxRail host details from inventory”

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Error message similar to the below are seen in /var/log/vmware/vcf/domainmanager/domainmanager.log: 

2020-10-19T13:13:36.886+0000 DEBUG [vcf_dm,8187b087ae8d6c92,ead7] [c.v.v.v.h.a.RemoveVxRailHostGetInventoryAction,dm-exec-3]  Fetching cluster 840a87cf-50e4-4e9e-b235-a1a8e02b4b8e VDSes

2020-10-19T13:13:37.139+0000 ERROR [vcf_dm,8187b087ae8d6c92,ead7] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-3]  [1QS5BE] POST_INTERNAL_SERVER_ERROR Internal server error

com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Internal server error
        at com.vmware.vxrail.vcf.hostmanager.action.RemoveVxRailHostGetInventoryAction.execute(RemoveVxRailHostGetInventoryAction.java:249)
        at com.vmware.vxrail.vcf.hostmanager.action.RemoveVxRailHostGetInventoryAction.execute(RemoveVxRailHostGetInventoryAction.java:42)

Environment

VMware Cloud Foundation 4.0.x

Cause

The ‘is_used_by_nsxt’ attribute in the ‘vds’ table in the SDDC Manager database is null after upgrading the environment from VCF 4.0.0 to 4.0.1 or higher. As a result, upgraded system fails to decide if a particular VDS is used for NSXT overlay traffic or not.

Resolution

This is a known issue and will be fixed in a future VMware Cloud Foundation release.

Workaround:

1. Download the attached is_used_by_nsxtFlagUpdate.sh script.
2. Using a file transfer utility, copy the script to /tmp on the SDDC Manager VM. 
3. Open a SSH to SDDC Manager VM and switch to root user.

su -

4. Change directory to /tmp and Run the script:

cd /tmp
./is_used_by_nsxtFlagUpdate.sh

4. Restart the failed task from SDDC Manager UI.