VMware Cloud Foundation UI not accessible after performing out of band upgrade of vCenter to version higher than 7.0.U2a

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:
UI server log fails to start UI with the ssh to vCenter timeout error as shows below:

2021-07-21T18:13:10.694+0000 ERROR [538cc30345854063, 5517448947d54f44] [services/pscUtils.js, init-pscs, attemptPSCInit:119] Caught error from await primaryPscInit
2021-07-21T18:13:10.694+0000 WARN [538cc30345854063, 5517448947d54f44] [services/pscUtils.js, init-pscs, attemptPSCInitWithRetry:170]
100.109: VError: PSC Initilization attempt "18" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Failed to init "primary PSC" with hostname: "<FQDN>": Remote ssh command timed out
    at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:85:5)
    at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:163:46)
Error Info: {"retryCount":18,"status":403,"errorModule":100,"errorCode":109}
caused by:
100.108: VError: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Failed to init "primary PSC" with hostname: "<FQDN>": Remote ssh command timed out
    at Object.initiatePscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:85:5)
    at attemptPSCInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:131:26)
    at async attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:158:17)
Error Info: {"errorModule":100,"errorCode":108}
caused by:
100.107: VError: Primary psc init failed and failover psc init also failed: Failed to init "primary PSC" with hostname: "<FQDN>": Remote ssh command timed out
    at Object.primaryPscInitError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:85:5)
    at attemptPSCInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:126:34)
    at async attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:158:17)
Error Info: {"errorModule":100,"errorCode":107}
caused by:
100.102: VError: Failed to init "primary PSC" with hostname: "<FQDN>": Remote ssh command timed out
    at Object.retrievePscLoginSecretError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:85:5)
    at retrievePscSecretAndInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:70:26)
Error Info: {"pscType":"primary PSC","hostname":"<FQDN>","status":403,"errorModule":100,"errorCode":102}
caused by:
Error: Remote ssh command timed out
    at /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:344:65
    at new Promise (<anonymous>)
    at init (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:343:24)
    at Object.reset (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:363:13)
    at initializeSSO (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:49:24)
    at retrievePscSecretAndInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:67:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

Cause

• vCenter drops support for ssh_rsa for ssh starting from vCenter version 7.0U2a.
• The SDDC Manager UI version 4.2.x OR lower relies on ssh-rsa based ssh to vCenter to fetch few meta-data information which will be broken in case of any vCenter version higher than 7.0.U2a.
• This causes the UI server to fail to come up after out of band upgrade of vCenter to 7.0U2a or higher version in a VCF setup.

Resolution

If you have VMware Cloud Foundation 4.2.1, please contact Broadcom Support.