Unable to remediate VCF Automation "vmware-system-user" password once expired.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• "vmware-system-user" password is not accepted in an SSH session for VCF Automation.
• Also in VCF Operations in Fleet Management > Passwords we see that the "vmware-system-user" password is showing as "Disconnected".

Environment

• VCF Automation 9.0.x

Cause

• Issue where the Secret is not aligned with the provided password.
• If you have manually reset the "vmware-system-user" from an SSH session or VM Console session to the VCF Automation 9 Appliance using:

  • passwd vmware-system-user

     

Resolution

1. SSH to the VCF Automation Appliance.

If you cannot log into the appliance using the "vmware-system-user" password, access the appliance console from vCenter using KB Resetting the root password on a Photon appliance - proceed with resolution steps 1 - 4.

2. Once you access the Shell you can run below to change the vmware-system-user password:

mount -o rw,remount /
passwd vmware-system-user

3. Reboot the VCF Automation appliance after changing the "vmware-system-user" password.

4. SSH to VCF Automation Primary appliance using "vmware-system-user" password and run the following command:

HASH=$(echo 'YOUR-NEW-PASSWORD' | vmsp passwd --password-stdin)    # Replace text [YOUR-NEW-PASSWORD] with your newly set vmware-system-user password.
SECRET=$(echo $HASH | base64 -w 0)
echo $SECRET

5. Take note of the value outputted from "$SECRET"

6. Access the SSH management using the following command:

sudo su
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get secrets -n vmsp-platform | grep ssh

7. Note the output of this command and replace the secret "vcf-mgmt-########-ssh-password-secret" with this output.

kubectl patch secret vcf-mgmt-########-ssh-password-secret -n vmsp-platform -p "{\"data\":{\"sshPassword\":\"OUTPUT-OF-$SECRET\"}}"
# Replace text [OUTPUT-OF-$SECRET] above with the $SECRET value saved earlier.

• This should provide the following output if successful:

secret/secret vcf-mgmt-########-ssh-password-secret patched

8. Confirm that user is synchronized properly in VCF Operations in Fleet Management > Passwords and shown as "Connected".

• Please wait up to 60 minutes before running "Update Password" again.

Note: If you find that day 2 operations is still failing in Fleet Manager for the VCF Automation component even after updating and then remediating password, please ensure that the correct password alias is selected when retrying the task.  If required please create a new password in the locker with the new password.  

Additional Information

With the upcoming 9.x release, a new Fleet LCM / VCF Operations component will be introduced. The updated VCF Ops will support password rotation and also provide the ability to configure a policy for passwords to never expire out of the box.