Deleting workload domain workflow failed at "Cleanup VC References in PSC" in VMware Cloud Foundation

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Workload domain deletion workflow failed on "Clean up the VC References in PSC" task.
• You see messages similar to the following in /home/vrack/vrm/logs/vrack-vrm-debug.log on the SDDC Manager Controller VM.

2018-07-13 19:33:05.660 [pool-23-thread-1] DEBUG [com.vmware.vrack.vrm.core.util.SshSession] Exception while taking SSH connection com.jcraft.jsch.JSchException: HostKey has been changed: rack-1-psc-2.vcf.xyz.local
com.jcraft.jsch.JSchException: HostKey has been changed: rack-1-psc-2.vcf.xyz.local
 

• You see Error message similar to the following on the SDDC Manger UI  >> System Status >> Workflow >> Sub Task >> Expand Cleanup VC Reference PSC

Cleaning up The reference  to VC rack-vc2-xyc.vcf.local from PSC at rack-psc-2-vcf-local

• You see messages similar to the following in /var/log/message on the PSC. [in this example it is rack-psc-2-vcf-local]

2018-07-13T19:31:05.607747+00:00 rack-1-psc-2 sshd[7760]: rexec line 124: Deprecated option RhostsRSAAuthentication
2018-07-13T19:31:05.631188+00:00 rack-1-psc-2 sshd[7760]: error: Received disconnect from 10.25.18.38 port 57720:3: com.jcraft.jsch.JSchException: HostKey has been changed: rack-1-psc-2.vcf.xyz.local [preauth]
2018-07-13T19:31:05.631531+00:00 rack-1-psc-2 sshd[7760]: Disconnected from 10.25.18.38 port 57720 [preauth]

 

Note: In this example, 10.25.118.38 is the SDDC Manager Controller VM IP address.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment

Environment

VMware Cloud Foundation 2.3.x
VMware Cloud Foundation 2.2.x

Resolution

1. SSH to the SDDC Manager controller VM as root.
2. Run the following command to login as "vrack" user account.

su - vrack

3. Change your working directory to  /home/vrack/.ssh

cd /home/vrack/.ssh

4. Creat a copy the known_hosts file for backup

cp -p known_hosts known_hosts-bak

5. Open the  known_hosts  file with a text editor

vi known_hosts

6. Press "i" to start editing.
7. Comment out the lines that contain  the  current SSH HostKeys for  the PSC FQDN and the PSC IP Address .

Example:
#172.30.0.33 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgXx7pKj4D63b6opRgddgddOpKQ78Vnxp9BmQzDbhYcIN5hmdYYCrIzF3OTs7D8hDvRXds0JjRH6fj05KuL9JZAgEELxHvIiQGvJ+A918NQoNpmW6dDSA07IaA2VZ442lyB9+BSXPnuuymFQ+fGtKVXArq3Qf+/MMlpBUKuywqQgOVQG52nLr4O79/5qdKVg/IpEqfya1hjOIGwnh6yLx5x33RxITii5gd5t9XrulL+MkE/ZS6tBvSQ59PCpwHCpIezh0bmgAxUs4FldbxxCOLI+6ynncAoxWq40my2QQqiZXNdQviCSfBR7N0DOuztbaSiWdggdgddJHiuSqo2QcRpx5ZevPmreIrfgdNIyZ

#psc-2.vcf.vxrack.local ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgXx7pKj4D63b6opRgddgddOpKQ78Vnxp9BmQzDbhYcIN5hmdYYCrIzF3OTs7D8hDvRXds0JjRH6fj05KuL9JZAgEELxHvIiQGvJ+A918NQoNpmW6dDSA07IaA2VZ442lyB9+BSXPnuuymFQ+fGtKVXArq3Qf+/MMlpBUKuywqQgOVQG52nLr4O79/5qdKVg/IpEqfya1hjOIGwnh6yLx5x33RxITii5gd5t9XrulL+MkE/ZS6tBvSQ59PCpwHCpIezh0bmgAxUs4FldbxxCOLI+6ynncAoxWq40my2QQqiZXNdQviCSfBR7N0DOuztbaSiWdggdgddJHiuSqo2QcRpx5ZevPmreIrfgdNIyZ

8. Save the file and exit by pressing Esc :wq!
9. Run the following commands from the SDDC Manager Controller VM to update the PSC ssh HostKey for the FQDN and IP Address.

ssh -oHostKeyAlgorithms='ssh-rsa' root@<psc ip>    
ssh -oHostKeyAlgorithms='ssh-rsa' root@<psc FQDN>

10. Restart the workload domain deletion workflow.