Pre-check Error - NTP time sync check failed for invalid known_hosts for xx.xx.xx.xx. This occurs due to invalid location of known_hosts. Check if the location of the known_hosts is valid and accessible. Retry the precheck

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• The Precheck fails on vCenter/PSC component​​ as below

Description Check the status of NTP health​​​​
Start Time May 3, 2022, 6:35:28 AM
End Time May 3, 2022, 6:35:28 AM
Health Status RED
Error Description reject HostKey: xx.xx.xx.xx
Impact High: Do not perform upgrade without addressing this issue.
Remediation NTP time sync check failed for invalid known_hosts for 10.207.36.13. This occurs due to invalid location of known_hosts. Check if the location of the known_hosts is valid and accessible. Retry the precheck.

• lcm.log located at /var/log/vmware/vcf/lcm will show similar entries

2022-05-03T10:05:51.081+0000 ERROR [0000000000000000,0000,precheckId=619a3b29-0dc9-474a-a32a-03bcefe2ce43,resourceType=PSC,resourceId=ba47a59c-b6e4-4131-9665-cdc52dc58b2f] [c.v.e.s.l.c.utils.SshCommandRunner,pool-2-thread-10] Command '[echo $0]' execution failed:com.jcraft.jsch.JSchException: fromBase64: invalid base64 data
       at com.jcraft.jsch.Util.fromBase64(Util.java:63)
       at com.jcraft.jsch.KnownHosts.setKnownHosts(KnownHosts.java:219)
       at com.jcraft.jsch.KnownHosts.setKnownHosts(KnownHosts.java:55)
       at com.jcraft.jsch.JSch.setKnownHosts(JSch.java:317)
       at com.vmware.evo.sddc.lcm.common.ssh.SshSessionFactory.getSessionPasswordAuth(SshSessionFactory.java:70)
       at com.vmware.evo.sddc.lcm.common.ssh.SshSessionFactory.getSession(SshSessionFactory.java:31)
       at com.vmware.evo.sddc.lcm.common.utils.RemoteCommandService.executeSshCommand(RemoteCommandService.java:34)
       at com.vmware.evo.sddc.lcm.common.utils.SshCommandRunner.executeWithResult(SshCommandRunner.java:34)
       at com.vmware.evo.sddc.lcm.primitive.common.utils.VmwPrimitiveUtils.runCmdOverSSH(VmwPrimitiveUtils.java:256)
       at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.CloudVMCommon.checkApplianceShellDefault(CloudVMCommon.java:888)
       at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.CloudVMCommon.initialize(CloudVMCommon.java:174)
       at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.CloudVMAuditor.initialize(CloudVMAuditor.java:38)
       at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterPatchImpl.initializeCloudVMAuditor(VCenterPatchImpl.java:350)
       at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterPatchImpl.precheck(VCenterPatchImpl.java:457)
       at com.vmware.evo.sddc.lcm.services.impl.UpgradePrecheckAsyncHandler$1.run(UpgradePrecheckAsyncHandler.java:195)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: null
2022-05-03T10:05:51.082+0000 ERROR [0000000000000000,0000,precheckId=619a3b29-0dc9-474a-a32a-03bcefe2ce43,resourceType=PSC,resourceId=ba47a59c-b6e4-4131-9665-cdc52dc58b2f] [c.v.e.s.l.p.i.vcenter.CloudVMCommon,pool-2-thread-10] VCenter upgrade finding current shell failed
2022-05-03T10:05:51.082+0000 ERROR [0000000000000000,0000,precheckId=619a3b29-0dc9-474a-a32a-03bcefe2ce43,resourceType=PSC,resourceId=ba47a59c-b6e4-4131-9665-cdc52dc58b2f] [c.v.e.s.l.p.i.vcenter.CloudVMCommon,pool-2-thread-10] retrying…
 

• Running below command shows

/var/log/vmware/vcf/lcm ]# grep "JSchException" lcm.log | sort -u
com.jcraft.jsch.JSchException: fromBase64: invalid base64 data
com.jcraft.jsch.JSchException: reject HostKey: xx.xx.xx.xx
com.jcraft.jsch.JSchException: reject HostKey: xx.xx.xx.xx
com.jcraft.jsch.JSchException: reject HostKey: xx.xx.xx.xx

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 3.10.0.0
VMware Cloud Foundation 3.10.x
VMware Cloud Foundation 3.11

Cause

 

Resolution

To resolve this issue it is required that  vCenters( mgmt, WLD) and PSCs  present ssh-rsa keys instead of ecdsa keys for SSH, and then add them to /home/vcf/.ssh/known_hosts.

For each of the PSCs/VCs, login as  root and:

1. Edit /etc/ssh/sshd_config
2. Uncomment the following line: #HostKey /etc/ssh/ssh_host_rsa_key
3. Restart the ssh daemon: systemctl restart sshd
4. Ensure the rsa key is being presented by running 'ssh -v localhost' (you do not need to log in).
5. On the SDDC Manager, remove any entries in /home/vcf/.ssh/known_hosts for the VCs and PSCs (IPs and/or FQDNs),
6. Add  the ssh-rsa keys back by running (As VCF user) ssh <fqdn> for each of the VCs and PSCs. You will be prompted to trust those keys, do so.
7. When all of the ssh-rsa keys are in the known_hosts file, restart LCM and perform the prechecks.