svc-vcf account disconnected from SDDC Manager

Products

• VMware SDDC Manager

Issue/Introduction

In the SDDC manager UI you will see a red banner stating that there are accounts for ESXi hosts that are currently disconnected.  You may also see Inventory Sync tasks and password validation workflows failing.

 

You may see errors similar to the following in the SDDC Manager /var/log/vmware/vcf/operationsmanager/operationsmanager.log 

 

Password validation failed:

DEBUG [vcf_om,6814c149049fd5904bf23b3ba66bbd23,6b81] [c.v.v.p.s.PasswordExpirationService,om-exec-2] Expiry retrieval status : UNKNOWN ,  Diagnostic message : {"errorCode":"PASSWORD_MANAGER_VALIDATE_ESXI_CREDENTIALS_FAILED","arguments":["ESXI-HOST-FQDN"],"errorMessage":"javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","referenceToken":"RT2U7G","remediationMessage":"Please verify that the account is active and is not locked, you might need to fix the workflow(s) for resources marked in error state. If the password of the account has expired, manually reset the password in the product and then perform a REMEDIATE operation in the SDDC Manager, to update its stored copy of the password."}

 

Inventory sync failed:

DEBUG [vcf_om,681586b01f2b82346bb7ea2f73b07ecc,8ce4] [c.v.v.v.s.i.VersionSyncServiceImpl,pool-8-thread-1] Validating the result of the inventory sync: {"entitiesToSync":["ESXI","NSXT_CLUSTER","VCENTER"],"entitiesToUpdate":[],"completedInventorySyncTasks":[{"entity":"ESXI","syncStatus":"FAILED","errors":[{"errorCode":"ESX_CONNECTION_ERROR","errorMessage":"Unable to connect to ESX host ESXI-HOST-FQDN.","errorType":"ERROR","cause":"com.vmware.vim.vmomi.client.exception.SslException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","remediation":"*****"}]},{"entity":"VCENTER","syncStatus":"SUCCEEDED","errors":[]},{"entity":"NSXT_CLUSTER","syncStatus":"SUCCEEDED","errors":[]}],"failedInventorySyncTasks":[{"entity":"ESXI","syncStatus":"FAILED","errors":[{"errorCode":"ESX_CONNECTION_ERROR","errorMessage":"Unable to connect to ESX host ESXI-HOST-FQDN.","errorType":"ERROR","cause":"com.vmware.vim.vmomi.client.exception.SslException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","remediation":"*****"}]}]}

Environment

VCF 4.x

VCF 5.x

Cause

This is caused by a mismatch of the ESXi host certificate  key in the SDDC manager database. Host Keys can be changed on a node for a variety of reasons, including but not limited to:

• Restoring from a backup
• Manual rebuild of the host
• Manual intervention to change the Host Key

As a result of this change key, SDDC Manager is unable to SSH into the node(s) in question to run through the attempted workflow due to a mismatch in what it expects the Host key to be vs what is being presented by the node.

 

Resolution

Update the host keys within SDDC using the fixHostkeys.py or fix_known_hosts.sh  from KB 316028

How to update the SSH host keys on the SDDC Manager