VMware Cloud Foundation VxRail Workload Domain deletion results in the SSH Keys for other domain VxRail Managers getting deleted from SDDC Manager known_hosts file.
search cancel

VMware Cloud Foundation VxRail Workload Domain deletion results in the SSH Keys for other domain VxRail Managers getting deleted from SDDC Manager known_hosts file.

book

Article ID: 318604

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:
LCM Precheck fails at:
Task: SSH VXM APPLIANCE CHECK
Description: Verify SSH Connection for VxRail manager Appliance.
Error Description: reject Hostkey: <VxRail Manager IP>

Environment

VMware Cloud Foundation 4.1
VMware Cloud Foundation 4.0.x

Cause

This problem will surface when different VxRail Managers belonging to different clusters will have the same SSH key, due to which the known_hosts file on SDDC Manager will have entries for different VxRail Manager but with the same key.

During the workload domain deletion, the SSH key for VxRail Manager belonging to the clusters being deleted, is removed from the known_hosts file by key, but as the key is the same for other VxRail Managers as well, their SSH Keys will also be deleted from the known_hosts file.

This will eventually cause an issue when LCM Precheck is executed post workload domain deletion. Due to the missing SSH Key for the existing VxRail managers, connectivity from SDDC Manager to the VxRail Manager will not work, hence the error will come up during LCM Precheck.

Resolution

This issue is fixed in VMware Cloud Foundation VCF 4.2 and subsequent releases.

Workaround:
To workaround this issue, you need to add the SSH Key for the existing VxRail Managers back to the SDDC Manager known_hosts file through the below steps: 

  1. Take a snapshot of the SDDC Manager VM through vCenter UI.
  2. SSH to SDDC Manager using vcf and then root. 
  3. Take a backup of the known_hosts files.
  cp -rf /home/vcf/.ssh/known_hosts /home/vcf/.ssh/known_hosts.BACKUP
 cp -rf /etc/vmware/vcf/commonsvcs/known_hosts /etc/vmware/vcf/commonsvcs/known_hosts.BACKUP 
  1. Next step is to check which VxRail manager entries are missing from the known_hosts file in SDDC Manager.
curl -X GET http://127.0.0.1:80/appliancemanager/ssh/knownHosts

Sample Output:

{"knownHosts":[{"host":"###.###.###.###","keyType":"ssh-rsa","key":"AAAAB3NzaC1yc2EAAAADAQABAAABAQDqQ+ou/+vjMYNIV0pHEIe6YKLKxGyWd+0DLbfkIceIJKWypFYXO4sGOxw3go6lXDG2Hs/Srp3CXNH16JdMnxKWYrlKY8cd+/s4k17IP60MEetiV83oEvYwO6RKUQQ7IomqELPkJjCYRXiZFzrGHwuXKA0pH0YLJxwcuTAH9Gj5OUYH3AlSNzpoGxRJVwm1o4HeEV6Y9WprX4rzqHLQvW9afD96UefOHob9dQIkh10WdotdstKgALA8DvLJtMmFugE5h+mm13OGsFJLaIbzWKgmha+UplwrSGMw/E1HVCorRQPaCVjCFuODTTholHKz95UehriVQL7LHGyTDrGplyQJ"},{"host":"###.###.###.###","keyType":"ssh-rsa","key":"AAAAB3NzaC1yc2EAAAADAQABAAABAQCc/MT03FH26aBNG7YRLU5DjyTKcdAAipxo4NlaLj/bwxZQgZlxnYOGaYjlDWPtykV971KOoJsHLNfRBud9OxQwySWxxhzUTsVChIImJyWYzN+kNimv61nsHzwH29OOhlrtZb2M11JUR4r/VvaWhI1o5L812D2coc2bHsrSUWd4oyZaqpaMlMWXV0MG9kH2mRBsCs561VmylwSSptae0mOBk+jXRjMBVlPov7nynTI03x+7XBeVTQabh94bGnw22Q5rr6w8O0dskxrhhRRmeMUMYJtDZUEY9vhxyN6tmy3Y2b1U2LDQ7+e2Dc6SuFTgbhGrgEIQyUbjdraGXW6VWYCl"}...
...
}
  1. From the above curl command output, check which VxRail manager entries are missing.
  2. For each VxRail Manager for which the SSH Key is missing from the known_hosts file, add the SSH key to the knwon_hosts files as below:
1-If the entry is missing from /home/vcf/.ssh/known_hosts, execute the below:
 ssh-keyscan -4 -t rsa VXRAIL_MANAGER_IP_OR_FQDN >> /home/vcf/.ssh/known_hosts

2-If the entry is missing from /etc/vmware/vcf/commonsvcs/known_hosts, execute the below:
ssh-keyscan -4 -t rsa VXRAIL_MANAGER_IP_OR_FQDN >> /etc/vmware/vcf/commonsvcs/known_hosts
 
  1. Check that the manually added SSH Key now comes up in the API output.
curl -X GET http://127.0.0.1:80/appliancemanager/ssh/knownHosts

 Sample Output:
  {"knownHosts":[{"host":"###.###.###.###","keyType":"ssh-rsa","key":"AAAAB3NzaC1yc2EAAAADAQABAAABAQDqQ+ou/+vjMYNIV0pHEIe6YKLKxGyWd+0DLbfkIceIJKWypFYXO4sGOxw3go6lXDG2Hs/Srp3CXNH16JdMnxKWYrlKY8cd+/s4k17IP60MEetiV83oEvYwO6RKUQQ7IomqELPkJjCYRXiZFzrGHwuXKA0pH0YLJxwcuTAH9Gj5OUYH3AlSNzpoGxRJVwm1o4HeEV6Y9WprX4rzqHLQvW9afD96UefOHob9dQIkh10WdotdstKgALA8DvLJtMmFugE5h+mm13OGsFJLaIbzWKgmha+UplwrSGMw/E1HVCorRQPaCVjCFuODTTholHKz95UehriVQL7LHGyTDrGplyQJ"},{"host":"###.###.###.###","keyType":"ssh-rsa","key":"AAAAB3NzaC1yc2EAAAADAQABAAABAQCc/MT03FH26aBNG7YRLU5DjyTKcdAAipxo4NlaLj/bwxZQgZlxnYOGaYjlDWPtykV971KOoJsHLNfRBud9OxQwySWxxhzUTsVChIImJyWYzN+kNimv61nsHzwH29OOhlrtZb2M11JUR4r/VvaWhI1o5L812D2coc2bHsrSUWd4oyZaqpaMlMWXV0MG9kH2mRBsCs561VmylwSSptae0mOBk+jXRjMBVlPov7nynTI03x+7XBeVTQabh94bGnw22Q5rr6w8O0dskxrhhRRmeMUMYJtDZUEY9vhxyN6tmy3Y2b1U2LDQ7+e2Dc6SuFTgbhGrgEIQyUbjdraGXW6VWYCl"},...
  ....
  ....
  {"host":"###.###.###.###","keyType":"ssh-rsa","key":"AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwRBInES1Bi0UZOIpCxvrQQTXeTCQY1w7rbrpIT0udzT+eTLvo1iOGKznRY3rxVW/labTqd1VkSa+sXNGvkWdN+lt7YgkAue1JTdejF3J0sQP9+LAxc4a8yFG3LiQFwimvOoTQlrLpEVXB9LiOLJPUnrdFvQQUD6EgnCLS1jv94Y/JtKYv6kb6wi924CSSaO08Yks450isMAFa8iYReti8kFrdVOOejpBfw0hhOpmPNmJA1IimcJk9KnABzp1nxYgWWIncRdiGVCsaf0ATuIdpUh3jMJ7A3/n+YrMut7tVL9qljQ3rjR0vz83VRjBAQVTmuVhCIBJ9ty/4hdRo9ij"}]}
  1. Execute Steps 6 and 7 for all the existing VxRail Managers for which the SSH Key is missing from the known_hosts file.
  2. Now retrigger the LCM Precheck which should now Succeed.
 
 
 
 



Powered by Wolken Software