"Failure while executing SDDC Manager drift app" error during SDDC Manager upgrade to 9.1

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

Upon reviewing the sddcmanager_migration_app_upgrade.log file, you may see errors similar to the following: 

{"errorCode":"VERSION_SYNC_FAILED","errorType":"ERROR","arguments":["Please ensure NSX manager NSX-MGR1-FQDN.com is running and reachable.; Please ensure NSX manager NSX-MGR2-FQDN.com is running and reachable."],"message":"Inventory version synchronization failed. Please try to fix the cause for the issue and retry the operation.","remediationMessage":"Please ensure the following: Please ensure NSX manager NSX-MGR1-FQDN.com is running and reachable.; Please ensure NSX manager NSX-MGR2-FQDN.com is running and reachable.","causes":[{"type":"com.vmware.evo.sddc.common.core.error.InvalidStateException","message":"Inventory version synchronization failed with: Unable to fetch NSX Manager resource versions with FQDN NSX-MGR1-FQDN.com to complete inventory sync.; Unable to fetch NSX Manager resource versions with FQDN NSX-MGR2-FQDN.com to complete inventory sync."}],"referenceToken":"1BJO1L"}]}

{"errorCode":"PASSWORD_MANAGER_RESOURCES_UNAVAILABLE","errorType":"ERROR","arguments":["[NSX-MGR3-FQDN.com]"],"message":"Resources [NSX-MGR3-FQDN.com] are not available/ready.","causes":[{"message":"Resources : [NSX-MGR3-FQDN.com]are not in ACTIVE state"}],"referenceToken":"B4PLCV"}]}]

Environment

VCF 9.1

Cause

This issue is caused by the NSX errors presented in the sddcmanager_migration_app_upgrade.log. 

Resolution

To resolve this issue, review and resolve each of the errors presented in the sddcmanager_migration_app_upgrade.log.

- To fix any disconnected NSX manager accounts in the SDDC inventory, reference the following KB articles:
NSX service account is disconnected in SDDC manager Password Management
Credential operations fail on NSX-T Components in SDDC Manager

- Verify and remove any stale NSX Manager references using the follwing KB article:
SDDC Manager UI displays "Failed to fect NSX information" following out-of-band NSX-T decommissioning

Once all of the NSX manager passwords are active, and all stale references have been removed, follow SDDC Manager 9.1 upgrade fails but UI shows upgrade as successful to start a new SDDC upgrade.