SSH to edge node fails with errors 'Unable to create jsch CLI session:' and 'reject HostKey:'

Products

• VMware Cloud Foundation

Issue/Introduction

Unable to rotate passwords for an edge cluster node, which fails with the following error:

2021-06-11T16:33:29.843+0000 ERROR [vcf_om,0000000000000000,0000] [c.v.evo.sddc.common.util.SshUtil,Thread-69] Unable to create jsch CLI session:
com.jcraft.jsch.JSchException: reject HostKey: <NSX Edge FQDN>

Environment

VMware Cloud Foundation 4.x

Cause

The SSH known_hosts keys are missing for the edge node or have been changed.
The edge node ID may be missing from the SDDC Manager database.

Resolution

1. Make backup of the '/home/vcf/.ssh/known_hosts' file
   cp /home/vcf/.ssh/known_hosts /home/vcf/.ssh/known_hosts.bak
   
   
2. Provide correct user for the file
    chown vcf:vcf /home/vcf/.ssh/known_hosts
   
   
3. Make sure the known_hosts file has correct permission
   ​​​​​​​ chmod 644 /home/vcf/.ssh/known_hosts
   
   
4. Provide correct user for the file
   ​​​​​​​ chown vcf:vcf /home/vcf/.ssh/known_hosts
   
   
5. Manually remove all entry of the edge node from “/home/vcf/.ssh/known_hosts” file
   
   
6. Login to SDDC via VCF user (Do not perform with root user)
   
   
7. Add proper entry into known host file by executing following command.
   ​​​​​​​ ssh-keyscan -4 -t rsa <IP ADDRESS OF THE EDGE NODE> >> /home/vcf/.ssh/known_hosts
​​​​​​​ ssh-keyscan -4 -t rsa <FQDN OF THE EDGE NODE> >> /home/vcf/.ssh/known_hosts
   
   
8. Verify the SSH entry is added to the known_hosts file
   
   
9. Login to the affected edge node with admin user and run the following:
    get user root status
 get user admin status
 get user audit status
   
      The output should show all user accounts to be in active state.
   
   
10. If any account is inactive, change the password to the same as what SDDC manager shows in lookup_passwords command. Once you change the password, the account should be marked as active.
     set user admin/audit/root password
    
    
11. Also validate if the user accounts are not expired. If expired, you can run the following to increase the limit
    set user admin password-expiration 120
    
    
12. Login to NSXT manager UI and retrieve the Edge node ID 
    NSXT UI -> System -> Fabric --> Nodes --> Edge Transport Nodes --> Select the affected edge node --> Copy the ID of the edge node displayed on the right panel (Copy the ID, not the External ID)
    
    
13. Run the following command > curl http://localhost/inventory/nsxt-edgeclusters | json_pp
           Also copy the "id" : "eb46f795-733d-40dd-9b3d-4a862cfb42d9" (Output may differ on different environment)
    
    Output:
    [
       {
          "clusterIds" : [
             "2933891c-8646-4390-a06b-def56edd5881"
          ],
          "nsxtClusterId" : "ffb65d34-0c65-43f0-ab45-d8b20082abb4",
          "name" : "abc-def-xyz",
          "edgeClusterNsxtId" : "7c772e54-5b16-417d-9e7a-96c9a842ad77",
          "status" : "ACTIVE",
          "nsxtEdgeNodes" : [
             {
                "hostName" : "<edge node1 FQDN>l",
                "id" : "d4bd41d5-ae15-49ba-a0e5-c8b20392ad65",
                "edgeNodeNsxtId" : "84f1def1-44d2-4c7d-9744-79259dd7de02",
                "managementIpAddress" : "<edge node1 IP>"
             },
             {
                "id" : "6029d679-2501-4d3a-843d-21a623b6fb12",
                "hostName" : "<edge node2 FQDN>l",
                "edgeNodeNsxtId" : "be6a24fa-6471-45d5-98d4-f4b3384cbad8",
                "managementIpAddress" : "<edge node1 IP>"
             },
             {
                "hostName" : "<edge node3 FQDN>l",
                "id" : "28a5554a-79d7-4708-beb4-adea8856fa1d",
                "managementIpAddress" : "<edge node1 IP>"
             }
          ],
          "id" : "eb46f795-733d-40dd-9b3d-4a862cfb42d9"
       }
    ]
             
    From the above output, edge node3 is missing the 'edgeNodeNsxtId'
    
    
14. Create json file and paste the above output having the correct edgeNodeNsxtId copied from step 12.
    NOTE: Make sure you do not copy the "[ ]" brackets from the above output into the json file.
    
    [ /home/vcf ]# touch test.json
    [ /home/vcf ]# vi test.json
    
    {
          "clusterIds" : [
             "2933891c-8646-4390-a06b-def56edd5881"
          ],
          "nsxtClusterId" : "ffb65d34-0c65-43f0-ab45-d8b20082abb4",
          "name" : "abc-def-xyz",
          "edgeClusterNsxtId" : "7c772e54-5b16-417d-9e7a-96c9a842ad77",
          "status" : "ACTIVE",
          "nsxtEdgeNodes" : [
             {
                "hostName" : "<edge node1 FQDN>l",
                "id" : "d4bd41d5-ae15-49ba-a0e5-c8b20392ad65",
                "edgeNodeNsxtId" : "84f1def1-44d2-4c7d-9744-79259dd7de02",
                "managementIpAddress" : "<edge node1 IP>"
             },
             {
                "id" : "6029d679-2501-4d3a-843d-21a623b6fb12",
                "hostName" : "<edge node2 FQDN>l",
                "edgeNodeNsxtId" : "be6a24fa-6471-45d5-98d4-f4b3384cbad8",
                "managementIpAddress" : "<edge node1 IP>"
             },
             {
                "hostName" : "<edge node3 FQDN>l",
                "id" : "28a5554a-79d7-4708-beb4-adea8856fa1d",
                "edgeNodeNsxtId" : "<copied ID from step 12>",
                "managementIpAddress" : "<edge node1 IP>"
             }
          ],
          "id" : "eb46f795-733d-40dd-9b3d-4a862cfb42d9"
       }
    
    
15. Run the command to sync the inventory:
    curl -X PUT -H "Content-Type: application/json" --data @test.json http://localhost/inventory/nsxt-edgeclusters/<copied ID from step 13>
    
    
16. Perform a new password rotation for the edge node users one by one in the below order.
    a) root
    b) admin
    c) audit
    
    

Additional Information

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-DB31B304-66A5-4516-9E55-2712D12B4F27.html

Impact/Risks:
None