Identifying hostkey mismatch of vCenter server in sddc Manager.

Products

• VMware SDDC Manager

Issue/Introduction

• This kb will help in identifying hostkey mismatch of vCenter server in sddc Manager.
• Steps can also be useful to investigate if SDDC manager UI stuck on Loading page.
• And can see below errors in /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log
  
  VError: PSC Initilization attempt "10" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: 500 - "\"Failed to establish SSH session to <vc_fqdn>\""
  at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError. js: 100:5)
  at processTicksAndRejections (node: internal/process/task_queues : 96: 5)
  at runMicrotasks (<anonymous>)
  at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils. js: 99:46)
  Error Info: {"retryCount": 10, "status": 403, "errorModule":100, "errorCode": 109}

Environment

• VMware Cloud Foundation 4.x

• VMware Cloud Foundation 5.x

Cause

When sddc manager fails ssh connection to vCenter server , hostkey mismatch is one of the cause behind inaccessible SDDC UI 

 

 

Resolution

To validate if hostkey is mismatch, follow below steps:

1. Get the current SSH key from vCenter
   ssh-keyscan -H <vCenter-FQDN> >> /tmp/current_vc_key
2. Compare with SDDC Manager’s known_hosts
   cat /home/vcf/.ssh/known_hosts | grep -i <vc_fqdn>
   
   or
   
   grep <vCenter-FQDN> /home/vcf/.ssh/known_hosts
   
   
3. Compare the fingerprints using output of below 2 commands:
   
   check the file generated in step 1:
   ssh-keygen -lf /tmp/current_vc_key     
   
   Check the exisiting entries:
   ssh-keygen -lf <(grep <vCenter-FQDN> /home/vcf/.ssh/known_hosts)
   
   Once compared, you should find mismatch in key.
   
   
4. Below outpout shows matching hostky in sddcmanager:
   
   root@sddc-manager [ /home/vcf ]# ssh-keygen -lf /tmp/current_vc_key
   
   256 SHA256:LDXXXXXXXXXXXXXXXXXXXXXXXXXXXXX (ED25519)
   3072 SHA256:b2a18QXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX (RSA)
   256 SHA256:+QKUszXXXXXXXXXXXXXXXXXXXXXXXXXXXXX (ECDSA)
   
   root@sddc-manager [ /home/vcf ]# ssh-keygen -lf <(grep <vc_fqdn> /home/vcf/.ssh/known hosts)
   
   256 SHA256:+QKUsz5XXXXXXXXXXXXXXXXXXXXXXXXX <vc_fqdn>(ECDSA)
   3072 SHA256:b2a18QXXXXXXXXXXXXXXXXXXXXXX <vc_fqdn> (RSA)
   
   
   
5. If the keys are mismatched, below kb will help to fix it:
   How to update the SSH host keys on the SDDC Manager