Async patch tool fails reporting the error: Retrieving NSX cluster upgrade status for domain failed due to NSX_RESOURCE_INFO_UNAVAILABLE:

Products

• VMware SDDC Manager

Issue/Introduction

Async patch tool fails with the below errors:

YYYY-MM-DD 17:04:14.161 [ERROR] Retrieving NSX cluster upgrade status for domain '####' failed due to NSX_RESOURCE_INFO_UNAVAILABLE: The NSX upgradable resource info for domain #### is unavailable. This is either due to an unhealthy NSX resource state or incomplete NSX inventory sync.

YYYY-MM-DD 17:04:19.277 [INFO ] Performing precheck - validate no in-progress upgrade
YYYY-MM-DD 17:04:19.379 [ERROR] Precheck failed due to the following errors:
YYYY-MM-DD 17:04:19.380 [ERROR] UNEXPECTED_EXCEPTION_THROWN
-- Error Message: Unexpected exception thrown: Retrieving NSX cluster upgrade status failed on the following domains: [####]
-- Remedy: Please contact GSS for further assistance.

YYYY-MM-DD 17:04:19.380 [ERROR] Error while initiating required functionality: INPUT_SPEC_WORKFLOW
YYYY-MM-DD 17:04:19.380 [ERROR] ENABLE_PATCH_PRECHECK_FAILED
-- Error Message: Enable Patch pre-checks failed due to above errors.
-- Remedy: Please resolve all above errors and rerun the AP tool to enable patches.

The following errors are seen in /var/log/vmware/vcf/lcm/lcm.log

YYYY-MM-DDT11:51:53.950+1000 ERROR [vcf_lcm,####,8668,auditId=####,resourceType=NSX_T_MANAGER,resourceId=NSXMANAGER-FQDN,name=NSXMANAGER-FQDN,] [c.v.e.s.l.p.impl.nsxt.NsxtAuditImpl,Scheduled-1] 

Error auditing NSX Cluster NSXMANAGER-FQDN, with exception {}
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSX Cluster from the Inventory

YYYY-MM-DDT11:51:53.950+1000 ERROR [vcf_lcm,####,8668,auditId=####,resourceType=NSX_T_MANAGER,resourceId=NSXMANAGER-FQDN,name=NSXMANAGER-FQDN] [c.v.v.c.n.s.c.c.ComplexHelpers,Scheduled-1] Exception occurred during NSX API invocation

java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized) => {
    messages = [],
    data = struct => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},
    errorType = UNAUTHORIZED
}

Environment

VMware Cloud Foundation 5.x

Cause

In the SDDC Manager UI -> Password Management -> The root account for NSX Manager was disconnected.

SDDC manager was unable to communicate with the NSX managers over SSH as

• The root user account password was not in sync.
• SSH was disabled on 2 out of the 3 NSX Managers.

Resolution

To resolve the issue,

• Reset the root account password, Credential operations fail on NSX Components in SDDC Manager
• Enable SSH on all NSX managers. KB 373427
• Remediate the root user account in SDDC Manager -> Password Management