SSH connection to vcenter fails during SDDC Manager Upgrade
search cancel

SSH connection to vcenter fails during SDDC Manager Upgrade

book

Article ID: 448134

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Upgrade failure due to VC ssh check from sddc manager during sddc manager upgrade 

SDDC Manager commonsvcs log
/var/log/vmware/vcf/commonsvc/vcf-commonsvcs.log

2026-05-23T11:35:23.901+0000 INFO [common,d40bd54768bd473f,8025] [c.v.e.s.c.s.a.r.CommonServicesSamlMetadataExchangeController,http-nio-127.0.0.1-7100-exec-11] Start getPscRootSshCredential(), Psc host: ESXI_Hostname
2026-05-23T11:35:23.904+0000 INFO [common,d40bd54768bd473f,8025] [c.v.e.s.i.s.PscInventoryServiceImpl,http-nio-127.0.0.1-7100-exec-11] Get Psc - b0a01398-3ab9-4c94-828a-12cb0c1a8a23
2026-05-23T11:35:23.912+0000 INFO [common,6a1190fb50eb4533b0d61c59fabd48b8,2f5d] [c.v.e.s.a.u.api.rest.SshController,http-nio-127.0.0.1-7100-exec-3] getSshKnownHostsConfiguration(): host = ESXI_Hostname
2026-05-23T11:35:23.913+0000 INFO [common,d40bd54768bd473f,8025] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-7100-exec-11] Mina sshd: Establishing SSH session to host: ESXI_Hostname
2026-05-23T11:35:24.044+0000 ERROR [common,0000000000000000,0000] [c.v.v.s.s.c.CustomKnownHostsServerKeyVerifier,sshd-SshClient[390c21bb]-nio2-thread-1] Server at ESXI_Hostname/ESXI_IP_Address:22 presented wrong key type: ecdsa-sha2-nistp256 and key: HOST_KEY
2026-05-23T11:35:24.045+0000 ERROR [common,d40bd54768bd473f,8025] [c.v.v.s.ssh.common.SshClientImpl,http-nio-127.0.0.1-7100-exec-11] Failed to open ssh connection to ESXI_Hostname in 30 seconds.
2026-05-23T11:35:24.045+0000 ERROR [common,d40bd54768bd473f,8025] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-7100-exec-11] error message: ESXI_Hostname: Server key did not validate
2026-05-23T11:35:24.045+0000 ERROR [common,d40bd54768bd473f,8025] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-7100-exec-11] Host key validation failed for host ESXI_Hostname
com.vmware.vcf.secure.ssh.common.SshClientException: ESXI_Hostname: Server key did not validate
at com.vmware.vcf.secure.ssh.common.SshClientImpl.connect(SshClientImpl.java:175)

Caused by: org.apache.sshd.common.SshException: Server key did not validate
at org.apache.sshd.client.session.AbstractClientSession.checkKeys(AbstractClientSession.java:645)
at org.apache.sshd.common.session.helpers.AbstractSession.handleKexMessage(AbstractSession.java:767)
at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:622)
at org.apache.sshd.common.session.helpers.AbstractSession.lambda$handleMessage$0(AbstractSession.java:546)
at org.apache.sshd.common.util.threads.ThreadUtils.runAsInternal(ThreadUtils.java:68)
at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:545)
at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1729)
at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:506)
at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:64)
at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:409)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:382)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:377)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:129)
at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:221)
at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:113)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
... 1 common frames omitted
2026-05-23T11:35:24.045+0000 ERROR [common,d40bd54768bd473f,8025] [c.v.e.s.c.s.a.r.CommonServicesSamlMetadataExchangeController,http-nio-127.0.0.1-7100-exec-11] Exception in shellAgnosticCommandExec for host ESXI_Hostname : com.vmware.vcf.secure.ssh.errors.VcfSshException: Failed to establish SSH session to ESXI_Hostname as the host key validation failed
2026-05-23T11:35:24.045+0000 ERROR [common,d40bd54768bd473f,8025] [c.v.e.s.c.s.a.r.CommonServicesSamlMetadataExchangeController,http-nio-127.0.0.1-7100-exec-11] Get error in getIDPMetadataFromVC
com.vmware.vcf.secure.ssh.errors.VcfSshException: Failed to establish SSH session to ESXI_Hostname as the host key validation failed

Environment

VMware Cloud Foundation 5.x

VMware Cloud Foundation 9.x

Cause

The vCenter hostname has a mix between upper case and lower case but sddc manager stores it as all lowercase.
During an upgrade, SDDC Manager is unable to SSH into the VCenter in question to run through the attempted workflow due to a mismatch in what it expects the Host key to be vs. what is being presented by the node.

Resolution

Engineering is working on a fix for this but until then please follow workaround 2 in this kb
Addressing vCenter FQDN and Hostname Case Mismatches

Configure Temporary PNID (exampleTemp.example.com):

  1. Update the VCSA Hosts File:
    1. Backup the original configuration:
       cp /etc/dnsmasq.conf /etc/dnsmasq.conf.original
    2. Edit dnsmasq.conf to include the hosts file:
       vi /etc/dnsmasq.conf
      Add the line:
      addn-hosts=/etc/dnsmasq.hosts
    3. Add the temporary PNID to /etc/dnsmasq.hosts:
      vi /etc/dnsmasq.hosts
      Add the line:
      <vCenter_IP_Address> exampleTemp.example.com exampleTemp
    4. Restart the dnsmasq service:
      systemctl restart dnsmasq.service
    5. Test the name resolution:
      nslookup exampleTemp.example.com 
      nslookup <vCenter_IP>
    6. Change the PNID to Temporary PNID in VAMI:
      1. Access the vCenter Appliance Management Interface: https://<vCenter_IP>:5480.
      2. Under Networking -> Edit, update the hostname to exampleTemp.example.com

        Note: It is recommended to add exampleTemp.example.com to the local hosts file of the web browser's PC to ensure accessibility to the VAMI interface.
    7. Reboot the vCenter Server and verify the changes.
    8. Remove Temporary PNID Configuration from Proxy DNS:
    9. Restore the original configuration:
      cp /etc/dnsmasq.conf.original /etc/dnsmasq.conf
    10. Delete the temporary hosts file:
      rm /etc/dnsmasq.hosts
    11. Restart the dnsmasq service:
      systemctl restart dnsmasq.service
    12. Change PNID to Destination PNID (vcsa02.example.com) in VAMI:
      1. Access the vCenter Appliance Management Interface: https://<vCenter_IP>:5480.
      2. Under Networking -> Edit, update the hostname to vcsa02.example.com
    13. Reboot the vCenter Server and verify the changes.

If that workaround is unable to be performed at this time you can change the lowercased information to uppercase to match the vCenter VM name in known hosts file. NOTE: This is a temporary workaround and should be changed via the steps above as soon as possible because there may be other Day N operations that fail.

Powered by Wolken Software