"New update cannot be started while Domain xxxxx or some of its components are not in active state" - error encountered when attempting VCF 9.0.2 to 9.1.0 upgrade of SDDC Manager

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• The upgrade of SDDC Manager from 9.0.2 to 9.1.0 is blocked due to a red banner error (see text in Article Title)
• In the SDDC Manager log file /var/log/vmware/vcf/lcm/lcm-debug.log, the following is observed -

java.io.FileNotFoundException: /nfs/vmware/vcf/nfs-mount/bundle/depot/local/index.v3 (Permission denied)

• In the SDDC Manager log file /var/log/vmware/vcf/lcm/lcm-debug.log, the following is observed -

Caused by: org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: "{<EOL>  "error_code" : 403,<EOL>  "module_name" : "common-services",<EOL>  "error_message" : "The credentials were incorrect or the account specified has been locked."<EOL>}"

Environment

VCF 9.1.0

Cause

The workflow is being blocked by a combination of factors - 

• incorrect directory ownerships/permissions on SDDC Manager
• stale status entries on the SDDC Manager database for Password Management workflows
• disconnected NSX service accounts on SDDC Manager
• NSX API security policy resulting in locked account

Resolution

• Address the incorrect directory ownerships/permissions on SDDC Manager using Article ID: 392923
• Address the stale status entries on the SDDC Manager database for Password Management workflows using Article ID: 379319
• Remediate the disconnected NSX service accounts on SDDC Manager using Article ID: 430558
  • Note - retrieve the NSX service account credentials from SDDC Manager see Article ID: 327195
  • Note - if the NSX API security policy has resulted in a locked account, to address this see Article ID: 314647
• Retry the upgrade workflow, it is expected that error will no longer be encountered