"vmware-system-user" status for VCF Automation and VCF Identity Broker Component in Fleet Management shows disconnected

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• From the VCF Fleet Management could see the account "vmware-system-user" for VCF Automation and VCFvIDB is in "Disconnected" status.
  
  
  
  
• Performing "Update Password" towards the VCF Automation "vmware-system-user" account from Fleet fails with below error
  

  Error Code: LCMVMSP10019
  com.vmware.vrealize.lcm.util.exception.SshAuthenticationFailureException: Cannot execute SSH commands. Verify the SSH login credentials.
  Caused by: java.lang.RuntimeException: Cannot create session for ssh://vmware-system-user@##.##.##.##
   at com.vmware.vrealize.lcm.util.SessionHolder.newSession(SessionHolder.java:57)
   at com.vmware.vrealize.lcm.util.SessionHolder.<init>(SessionHolder.java:37)
   at com.vmware.vrealize.lcm.util.SshUtils.execute(SshUtils.java:694)
   ... 8 more
  Caused by: com.jcraft.jsch.JSchException: Auth fail for methods 'publickey,password,keyboard-interactive'
   at com.jcraft.jsch.Session.connect(Session.java:501)
   at com.vmware.vrealize.lcm.util.SessionHolder.newSession(SessionHolder.java:53)
   ... 10 more

Environment

VCF Operations 9.0
VCF Automation 9.0
VCF vIDB 9.0

Cause

The issue occurs because the "vmware-system-user" account password expires.
The expired "vmware-system-user" is preventing VCF Fleet Management from establishing necessary SSH connections to update the VCF Automation and VCF vIDB password.

Resolution

Follow the below steps to resolve this issue:

• Access the VCF Automation and VCF vIDB VM GNU-GRUB following the below steps:
  
      1. Navigate to the VM in the vSphere web client.
      2. Click on : ACTIONS  > Edit Settings on the VM, or right-click on the VM in the Inventory view for Edit Settings.
      3. Go to "VM Options" -> expand "Boot Options" and set the "Boot Delay" to 5000 milliseconds. This will make it easier to enter the GNU-GRUB on the next start of the VM.
      4. Enter the GRUB mode as mentioned in KB325916
      5. Check the status of the vmware-system-user's password expiration:
               chage -l vmware-system-user

  6.Set the vmware-system-user to never expire:
      chage -m 0 -M -1 vmware-system-user

  7. Confirm that the vmware-system-user is no longer expired:
      chage -l vmware-system-user

  8. Reset the "vmware-system-user" password:
              passwd vmware-system-user

            Note : Make sure to retain the same password     

             9. Reboot the VM by running:
         reboot -f

• Once the above task is completed, access VCF Operations UI >> Fleet Management >> Passwords.
  Select the "vmware-system-user" account and perform "Update Password".
  
  
• Trigger inventory sync towards the VCF Automation and VCF vIDB component.