known_hosts
files by using a script to remove the existing erroneous entries, and updating them with new ones.Validation, Deployment, PreChecks and other workflows on the SDDC Manager are failing with errors similar to:
... Unable to create jsch CLI session ...
... com.jcraft.jsch.JSchException: reject HostKey: [Node_FQDN_or_IP] ...
at com.jcraft.jsch.Session.checkHost(Session.java:789)
at com.jcraft.jsch.Session.connect(Session.java:345)
SSH attempts to PSC/VC can fail with the following errors:
100.109: VError: PSC Initilization attempt "9" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: 500 - "\"Failed to establish SSH session to <VC_FQDN>\""
at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:100:5)
at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:108:46)
SSH attempts to ESXi/NSX/vRealize/WS1 Nodes can fail with the following errors:
ERROR [vcf_om,0000000000000000,0000] [c.v.evo.sddc.common.util.SshUtil,Thread-69] Unable to create jsch CLI session:
com.jcraft.jsch.JSchException: reject HostKey: <NSX_Node_FQDN>
VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
Host Keys can be changed on a node for a variety of reasons, including but not limited to:
As a result of this change key, SDDC Manager is unable to SSH into the node(s) in question to run through the attempted workflow due to a mismatch in what it expects the Host key to be vs what is being presented by the node.
The fixHostKeys.py is the newest script to run, and recommended one to use.
Note: This script works in a FIPS enabled environment.
fixHostkeys.py
script attached to the KB python fixHostKeys.py --node <FQDN of a specific node>
Logs are written to:/var/log/vmware/vcf/fixHostKeys.log
In case the fixHostKeys.py
script fails to detect and update the host keys, then fallback to fix_known_hosts.sh
bash script.
Note: fix_known_hosts.sh
does not work in a FIPS enabled environment.
fix_known_hosts.sh
script attached to the KB:
/root/.ssh/known_hosts
/etc/vmware/vcf/commonsvcs/known_hosts
/home/vcf/.ssh/known_hosts
/opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts
Since this is a not a major change, the risk is minimal. That being said, since entries are being removed and added, a snapshot of the SDDC Manager is highly recommended, in the event we need to revert to a state from before the script was run.
bash: ./fix_known_hosts.sh: /bin/bash^M: bad interpreter: No such file or directory
This error is caused by DOS carriage returns added to the script when copying from a Windows based text editor. To resolve this problem, run the following command and rerun the script:
sed -i -e 's/\r$//' fix_known_hosts.sh