SDDC Manager password remediation fails with "Failed to execute command, error : SSH: Failed to establish SSH session"
search cancel

SDDC Manager password remediation fails with "Failed to execute command, error : SSH: Failed to establish SSH session"

book

Article ID: 408941

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

SDDC Manager Flags Account(s) as Disconnected After vIDM patch update.

We might see an error saying "accounts have been disconnected. Visit Password Management page to take action."

root accounts for vIDM are disconnected as shown below

 

 

Password Remediation fails with the error below.

 

 

 

Operations Manager log shows below errors.

Log file: /var/log/vmware/vcf/operationsmanager/operationsmanager.log

YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-14] ssh password test started for <node_fqdn>
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-5] =====> Testing with actual account: root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-14] =====> Testing with actual account: root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.h.LinuxHostHelperService,om-exec-14] Attempting to connect to host : <node_fqdn>, using username : root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.h.LinuxHostHelperService,om-exec-5] Attempting to connect to host : <node_fqdn>, using username : root
YYYY-MM-DDThh:mm:ss.###+0000 INFO  [vcf_om,##########################,####] [c.v.v.p.s.PasswordValidationService,om-exec-27] No inprogress password manager operation, so skipping resource status check
YYYY-MM-DDThh:mm:ss.###+0000 INFO  [vcf_om,##########################,####] [c.v.v.p.s.PasswordValidationService,om-exec-9] No inprogress password manager operation, so skipping resource status check
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-9] ssh password test started for <node_fqdn>
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.u.c.SshPasswordChanger,om-exec-9] =====> Testing with actual account: root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.p.h.LinuxHostHelperService,om-exec-9] Attempting to connect to host : <node_fqdn>, using username : root
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.c.s.Secu####tyConfigurationServiceImpl,om-exec-27] Security config ret####eved {"fipsMode":false}
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.t.DynamicTrustManager,reactor-http-nio-3] Checking validity of certificate chain CN=<node_fqdn>, OU=####, O=####, L=####, ST=####, C=####,CN=####, O=####,CN=#### ####, O=####
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.t.DynamicTrustManager,reactor-http-nio-3] Certificate chain CN=<node_fqdn>, OU=####, O=####, L=####, ST=####, C=####,CN=####, O=####,CN=#### ####, O=#### is valid
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,########################,####] [c.v.v.s.c.s.Secu####tyConfigurationServiceImpl,om-exec-14] Security config ret####eved {"fipsMode":false}
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.c.s.Secu####tyConfigurationServiceImpl,om-exec-5] Security config ret####eved {"fipsMode":false}
YYYY-MM-DDThh:mm:ss.###+0000 DEBUG [vcf_om,##########################,####] [c.v.v.s.c.s.Secu####tyConfigurationServiceImpl,om-exec-9] Security config ret####eved {"fipsMode":false}
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,####] [c.v.evo.sddc.common.util.SshUtil,om-exec-9] Unable to create jsch CLI session:
com.jcraft.jsch.JSchException: Algorithm negotiation fail
...
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,9dd5] [c.v.e.s.c.u.c.SshCommandExecuter,om-exec-9] Could not connect to the SSH server @ <node_fqdn> for configuration.
com.jcraft.jsch.JSchException: Algorithm negotiation fail
...
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,9dd5] [c.v.v.p.h.LinuxHostHelperService,om-exec-9] Exception when testing host credentials
com.vmware.evo.sddc.common.util.command.CommandExecuterException: SSH: Failed to establish SSH session to <node_fqdn>
...
Caused by: com.jcraft.jsch.JSchException: Algorithm negotiation fail
...
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,9dd5] [c.v.v.p.u.c.SshPasswordChanger,om-exec-9] Failed to execute command, error : SSH: Failed to establish SSH session to <node_fqdn>
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,########################,####] [c.v.evo.sddc.common.util.SshUtil,om-exec-14] Unable to create jsch CLI session:
com.jcraft.jsch.JSchException: Algorithm negotiation fail
...
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,##########################,9dd5] [c.v.v.p.u.c.SshPasswordChanger,om-exec-9] Unable to login to <node_fqdn> with username root got from CSS
YYYY-MM-DDThh:mm:ss.###+0000 ERROR [vcf_om,########################,####] [c.v.e.s.c.u.c.SshCommandExecuter,om-exec-14] Could not connect to the SSH server @ <node_fqdn> for configuration.
com.jcraft.jsch.JSchException: Algorithm negotiation fail

 

 

Environment

SDDC 5.x

Cause

Inspecting /etc/ssh/sshd_config on the vIDM nodes revealed this :

HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

HostkeyAlgorithms -ssh-rsa

 

This effectively disables ssh-rsa, leaving no matching host key algorithm that SDDC Manager trusts

 

Resolution

Please follow below instructions to resolve the issue:

1. Take a snapshot of SDDC manager.

2. Reconfigure SSH on vIDM Nodes

3. Edit /etc/ssh/sshd_config using VI command to appear as below.


#HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

HostkeyAlgorithms -ssh-rsa

 

After making the changes, it should look as shown below.

4. Restart SSH:
systemctl restart sshd

 

 

5. Re-Trust Host Keys in SDDC Manager

Follow KB 316028 and download the fixHostKeys.py script. Store it on the SDDC Manager and run the below command.

# python fixHostKeys.py --node <vIDM-FQDN>

 

6. Remediation password on SDDC with the current Root password for vIDM node. This will re-establish the connection.

Powered by Wolken Software