Pre-check Error - NTP time sync check failed for invalid known_hosts for xx.xx.xx.xx. This occurs due to invalid location of known_hosts. Check if the location of the known_hosts is valid and accessible. Retry the precheck
book
Article ID: 345253
calendar_today
Updated On:
Products
VMware Cloud Foundation
Issue/Introduction
Symptoms:
The Precheck fails on vCenter/PSC component as below
Description Check the status of NTP health Start Time May 3, 2022, 6:35:28 AM End Time May 3, 2022, 6:35:28 AM Health Status RED Error Description reject HostKey: xx.xx.xx.xx Impact High: Do not perform upgrade without addressing this issue. Remediation NTP time sync check failed for invalid known_hosts for 10.207.36.13. This occurs due to invalid location of known_hosts. Check if the location of the known_hosts is valid and accessible. Retry the precheck.
lcm.log located at /var/log/vmware/vcf/lcm will show similar entries
2022-05-03T10:05:51.081+0000 ERROR [0000000000000000,0000,precheckId=619a3b29-0dc9-474a-a32a-03bcefe2ce43,resourceType=PSC,resourceId=ba47a59c-b6e4-4131-9665-cdc52dc58b2f] [c.v.e.s.l.c.utils.SshCommandRunner,pool-2-thread-10] Command '[echo $0]' execution failed:com.jcraft.jsch.JSchException: fromBase64: invalid base64 data at com.jcraft.jsch.Util.fromBase64(Util.java:63) at com.jcraft.jsch.KnownHosts.setKnownHosts(KnownHosts.java:219) at com.jcraft.jsch.KnownHosts.setKnownHosts(KnownHosts.java:55) at com.jcraft.jsch.JSch.setKnownHosts(JSch.java:317) at com.vmware.evo.sddc.lcm.common.ssh.SshSessionFactory.getSessionPasswordAuth(SshSessionFactory.java:70) at com.vmware.evo.sddc.lcm.common.ssh.SshSessionFactory.getSession(SshSessionFactory.java:31) at com.vmware.evo.sddc.lcm.common.utils.RemoteCommandService.executeSshCommand(RemoteCommandService.java:34) at com.vmware.evo.sddc.lcm.common.utils.SshCommandRunner.executeWithResult(SshCommandRunner.java:34) at com.vmware.evo.sddc.lcm.primitive.common.utils.VmwPrimitiveUtils.runCmdOverSSH(VmwPrimitiveUtils.java:256) at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.CloudVMCommon.checkApplianceShellDefault(CloudVMCommon.java:888) at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.CloudVMCommon.initialize(CloudVMCommon.java:174) at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.CloudVMAuditor.initialize(CloudVMAuditor.java:38) at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterPatchImpl.initializeCloudVMAuditor(VCenterPatchImpl.java:350) at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterPatchImpl.precheck(VCenterPatchImpl.java:457) at com.vmware.evo.sddc.lcm.services.impl.UpgradePrecheckAsyncHandler$1.run(UpgradePrecheckAsyncHandler.java:195) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ArrayIndexOutOfBoundsException: null 2022-05-03T10:05:51.082+0000 ERROR [0000000000000000,0000,precheckId=619a3b29-0dc9-474a-a32a-03bcefe2ce43,resourceType=PSC,resourceId=ba47a59c-b6e4-4131-9665-cdc52dc58b2f] [c.v.e.s.l.p.i.vcenter.CloudVMCommon,pool-2-thread-10] VCenter upgrade finding current shell failed 2022-05-03T10:05:51.082+0000 ERROR [0000000000000000,0000,precheckId=619a3b29-0dc9-474a-a32a-03bcefe2ce43,resourceType=PSC,resourceId=ba47a59c-b6e4-4131-9665-cdc52dc58b2f] [c.v.e.s.l.p.i.vcenter.CloudVMCommon,pool-2-thread-10] retrying…
Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.
Environment
VMware Cloud Foundation 3.10.0.0 VMware Cloud Foundation 3.10.x VMware Cloud Foundation 3.11
Cause
Resolution
To resolve this issue it is required that vCenters( mgmt, WLD) and PSCs present ssh-rsa keys instead of ecdsa keys for SSH, and then add them to /home/vcf/.ssh/known_hosts.
For each of the PSCs/VCs, login as root and:
Edit /etc/ssh/sshd_config
Uncomment the following line: #HostKey /etc/ssh/ssh_host_rsa_key
Restart the ssh daemon: systemctl restart sshd
Ensure the rsa key is being presented by running 'ssh -v localhost' (you do not need to log in).
On the SDDC Manager, remove any entries in /home/vcf/.ssh/known_hosts for the VCs and PSCs (IPs and/or FQDNs),
Add the ssh-rsa keys back by running (As VCF user) ssh <fqdn> for each of the VCs and PSCs. You will be prompted to trust those keys, do so.
When all of the ssh-rsa keys are in the known_hosts file, restart LCM and perform the prechecks.