VCF 5.2.x on VxRail 8.0.x upgrade fails with error: "VxRail component upgrade failed with error: Password expiry retrieval failed for example.vrm.local"

Products

• VMware SDDC Manager

Issue/Introduction

• All ESXi and components appear to be on latest version from vCenter, and VxRail UI, however in SDDC manager the old versions might be seen due to the failed upgrade workflow. 

• Password for components is not expired or locked.
• In the /var/log/vmware/vcf/operationsmanager/operationsmanager.log you errors similar to below might be found: 

YYYY-MM-DDThh:mm:ss.558+0000 ERROR [vcf_om,0000000000000000000,0000] [c.v.v.p.s.PasswordValidationService,om-exec-16] Password expiry retrieval is failed for entity: VXRAIL_MANAGER, credential type: SSH, address: example.vrm.local
com.vmware.evo.sddc.common.util.command.CommandExecuterException: SSH: Failed to establish SSH session to example.vrm.local

YYYY-MM-DDThh:mm:ss.563+0000 DEBUG [vcf_om,0000000000000000,0000] [c.v.v.p.s.PasswordExpirationService,om-exec-2] Expiry retrieval status : UNKNOWN ,  Diagnostic message : {"errorCode":"PASSWORD_MANAGER_RETRIEVE_PASSWORD_EXPIRY_FAILED","arguments":["example.vrm.local"],"errorMessage":"SSH: Failed to establish SSH session toexample.vrm.local","referenceToken":"AAA000","remediationMessage":"Please verify that the account is active and is not locked, you might need to fix the workflow(s) for resources marked in error state. If the password of the account has expired, manually reset the password in the product and then perform a REMEDIATE operation in the SDDC Manager, to update its stored copy of the password."}

Environment

VMware Cloud Foundation 5.2.x

VxRail 8.0.3

Cause

SSH Access affected due to Host keys not matching and SDDC is unable to SSH to the node(s) in question to apply/complete the workflow it is running

Resolution

Please follow following KB article steps to resolve the Host key access issue and retry the upgrade workflow:

How to update the SSH host keys on the SDDC Manager