Workspace One Access nodes in password management are in a Disconnected state in the SDDC Manager UI

Products

• VMware Cloud Foundation

Issue/Introduction

Password remediations from the SDDC Manager fail for Workspace One Access nodes:

• vIDM patch CSP-97577 has been applied to the vIDM environment.
  
  
• In the SDDC Password Manager UI, the vIDM nodes are in a Disconnected state. The current passwords in use are valid and not locked. 
  
  
• Operationsmanager logs show the below errors on SSH connectivity failure: Algorithm negotiation fail

  Log File: /var/log/vmware/vcf/operationsmanager/operationsmanager.log:
  
  YYYY-MM-DDTHH:MM:SS.528+0000 DEBUG [vcf_om,689b23####,24f9] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-6] Remediation required: Performing test before remediation for username: root
  YYYY-MM-DDTHH:MM:SS.529+0000 DEBUG [vcf_om,689b23####,24f9] [c.v.v.p.u.c.SshPasswordChanger,om-exec-6] ssh password test started for ######.###.###
  YYYY-MM-DDTHH:MM:SS.529+0000 DEBUG [vcf_om,689b23####,24f9] [c.v.v.p.u.c.SshPasswordChanger,om-exec-6] =====> Testing with actual account: root
  YYYY-MM-DDTHH:MM:SS.529+0000 DEBUG [vcf_om,689b23####,24f9] [c.v.v.p.h.LinuxHostHelperService,om-exec-6] Attempting to connect to host : ######.###.###, using username : root
  YYYY-MM-DDTHH:MM:SS.610+0000 DEBUG [vcf_om,689b23####,24f9] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,om-exec-6] Security config retrieved {"fipsMode":false}
  YYYY-MM-DDTHH:MM:SS.747+0000 ERROR [vcf_om,689b23####,24f9] [c.v.evo.sddc.common.util.SshUtil,om-exec-6] Unable to create jsch CLI session:
  com.jcraft.jsch.JSchException: Algorithm negotiation fail
          at com.jcraft.jsch.Session.receive_kexinit(Session.java:590)
          ...
          at java.base/java.lang.Thread.run(Thread.java:840)
  YYYY-MM-DDTHH:MM:SS748+0000 ERROR [vcf_om,689b23####,24f9] [c.v.e.s.c.u.c.SshCommandExecuter,om-exec-6] Could not connect to the SSH server @ ######.###.### for configuration.
  com.jcraft.jsch.JSchException: Algorithm negotiation fail
          at com.jcraft.jsch.Session.receive_kexinit(Session.java:590)

   

Environment

VCF 5.x

Cause

The RSA hostkey algorithm has been deprecated in vIDM.

The vIDM agent nodes are disconnected as SDDC Manager tries to use the RSA HostKey to communicate with the vIDM VMs. 

Resolution

To resolve the issue:

1. Update the vIDM SSH server configuration to use ecdsa and ed25519 HostKeys
   
   
   1. Backup the sshd_config files in the vIDM nodes:
      

      cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

   2. Edit /etc/ssh/sshd_config file on all vIDM nodes and modify as listed below. This will change the SSHD configuration to use ecdsa and ed25519 algorithms for HostKeys.

      #HostKey /etc/ssh/ssh_host_rsa_key
      HostKey /etc/ssh/ssh_host_ecdsa_key
      HostKey /etc/ssh/ssh_host_ed25519_key

   3. Restart SSHD using the below command:
      

      systemctl restart sshd

2. Take a Snapshot of SDDC Manager.
   
   
3. Use the fixHostkeys.py script from KB article, How to update the SSH host keys on the SDDC Manager
   For eg. python fixHostKeys.py --node <FQDN of a specific node>


4. Remediate the passwords for the vIDM agent nodes using the current password. See, Remediate Passwords.