After patching vIDM, SDDC Manager password remediation fails with "Cause: Failed to execute command, error : SSH: Failed to establish SSH session"
search cancel

After patching vIDM, SDDC Manager password remediation fails with "Cause: Failed to execute command, error : SSH: Failed to establish SSH session"

book

Article ID: 408941

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • SDDC Manager flags accounts as disconnected after a vIDM patch update
  • You might see an error stating "accounts have been disconnected. Visit Password Management page to take action."
  • The root accounts for vIDM are disconnected as shown below:

  • Password Remediation fails with the following error:

  • SDDC Manager - /var/log/vmware/vcf/operationsmanager/operationsmanager.log shows the following errors:
    YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-14] ssh password test started for <node_fqdn>
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-5] =====> Testing with actual account: root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-14] =====> Testing with actual account: root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.h.LinuxHostHelperService,om-exec-14] Attempting to connect to host : <node_fqdn>, using username : root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.h.LinuxHostHelperService,om-exec-5] Attempting to connect to host : <node_fqdn>, using username : root
YYYY-MM-DDThh:mm:ss.###+0000 INFO  [vcf_om,##########################,####] [c.v.v.p.s.PasswordValidationService,om-exec-27] No inprogress password manager operation, so skipping resource status check
YYYY-MM-DDThh:mm:ss.###+0000 INFO  [vcf_om,##########################,####] [c.v.v.p.s.PasswordValidationService,om-exec-9] No inprogress password manager operation, so skipping resource status check
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-9] ssh password test started for <node_fqdn>
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-9] =====> Testing with actual account: root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.h.LinuxHostHelperService,om-exec-9] Attempting to connect to host : <node_fqdn>, using username : root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.c.s.Secu####tyConfigurationServiceImpl,om-exec-27] Security config ret####eved {"fipsMode":false}
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.t.DynamicTrustManager,reactor-http-nio-3] Checking validity of certificate chain CN=<node_fqdn>, OU=####, O=####, L=####, ST=####, C=####,CN=####, O=####,CN=#### ####, O=####
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.t.DynamicTrustManager,reactor-http-nio-3] Certificate chain CN=<node_fqdn>, OU=####, O=####, L=####, ST=####, C=####,CN=####, O=####,CN=#### ####, O=#### is valid
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,########################,####] [c.v.v.s.c.s.Secu####tyConfigurationServiceImpl,om-exec-14] Security config ret####eved {"fipsMode":false}
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.c.s.Secu####tyConfigurationServiceImpl,om-exec-5] Security config ret####eved {"fipsMode":false}
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.c.s.Secu####tyConfigurationServiceImpl,om-exec-9] Security config ret####eved {"fipsMode":false}
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,####] [c.v.evo.sddc.common.util.SshUtil,om-exec-9] Unable to create jsch CLI session:
com.jcraft.jsch.JSchException: Algorithm negotiation fail
...
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,9dd5] [c.v.e.s.c.u.c.SshCommandExecuter,om-exec-9] Could not connect to the SSH server @ <node_fqdn> for configuration.
com.jcraft.jsch.JSchException: Algorithm negotiation fail
...
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,9dd5] [c.v.v.p.h.LinuxHostHelperService,om-exec-9] Exception when testing host credentials
com.vmware.evo.sddc.common.util.command.CommandExecuterException: SSH: Failed to establish SSH session to <node_fqdn>
...
Caused by: com.jcraft.jsch.JSchException: Algorithm negotiation fail
...
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,9dd5] [c.v.v.p.u.c.SshPasswordChanger,om-exec-9] Failed to execute command, error : SSH: Failed to establish SSH session to <node_fqdn>
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,########################,####] [c.v.evo.sddc.common.util.SshUtil,om-exec-14] Unable to create jsch CLI session:
com.jcraft.jsch.JSchException: Algorithm negotiation fail
...
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,9dd5] [c.v.v.p.u.c.SshPasswordChanger,om-exec-9] Unable to login to <node_fqdn> with username root got from CSS
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,########################,####] [c.v.e.s.c.u.c.SshCommandExecuter,om-exec-14] Could not connect to the SSH server @ <node_fqdn> for configuration.
com.jcraft.jsch.JSchException: Algorithm negotiation fail

Environment

SDDC 5.x

Cause

Inspecting /etc/ssh/sshd_config on the vIDM nodes reveal the following configuration:

HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

HostkeyAlgorithms -ssh-rsa

The above configuration disables ssh-rsa, leaving no matching host key algorithm that SDDC Manager trusts.

Resolution

  1. Take a snapshot of the SDDC Manager.
  2. Reconfigure SSH on the vIDM nodes.
    1. Edit the /etc/ssh/sshd_config file:

      vi /etc/ssh/sshd_config

    2. Navigate to the following lines: 
      HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

HostkeyAlgorithms -ssh-rsa
    3. Edit the lines to match the following: 
      #HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

HostkeyAlgorithms -ssh-rsa
    4. Restart SSH on the vIDM node:

      systemctl restart sshd

  3. Re-trust the host keys in SDDC Manager.
    1. Follow KB 316028 to download the fixHostKeys.py script. Store it on the SDDC Manager and run the following command:
      python fixHostKeys.py --node <vIDM-FQDN>

  4. Remediate the password in the SDDC Manager UI using the current root password for the vIDM node to re-establish the connection.
Powered by Wolken Software