Unable to remediate VCF Automation "vmware-system-user" password once expired.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• "vmware-system-user" password is not accepted in an SSH session for VCF Automation.
• Also in VCF Operations in Fleet Management > Passwords we see that the "vmware-system-user" password is showing as "Disconnected".

Environment

• VCF Automation 9.0.x

Cause

• Issue where the Secret is not aligned with the provided password.
• If you have manually reset the "vmware-system-user" from an SSH session or VM Console session to the VCF Automation 9 Appliance using:

  • passwd vmware-system-user

     

Resolution

• SSH to the VCF Automation Appliance.

If you cannot log into the appliance using the "vmware-system-user" password, reset it from the VM Console in vCenter with the following KB:
Resetting the root password on a Photon appliance

• Once you can access the Shell you can run:

passwd vmware-system-user

• We can now SSH to VCF Automation Primary appliance and run the following:

HASH=$(echo 'YOUR-NEW-PASSWORD' | vmsp passwd --password-stdin)    # Replace text [YOUR-NEW-PASSWORD] with your newly set vmware-system-user password.
SECRET=$(echo $HASH | base64 -w 0)
echo $SECRET

• Take note of the value outputted from "$SECRET"
• We need to access the SSH management 

sudo su
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get secrets -n vmsp-platform | grep ssh

• Note the output of this command and replace the secret "vcf-mgmt-########-ssh-password-secret" with this output.

kubectl patch secret vcf-mgmt-########-ssh-password-secret -n vmsp-platform -p "{\"data\":{\"sshPassword\":\"OUTPUT-OF-$SECRET\"}}"
# Replace text [OUTPUT-OF-$SECRET] above with the $SECRET value saved earlier.

• This should provide the following output if successful:

secret/secret vcf-mgmt-########-ssh-password-secret patched

• This should then be synchronized properly in VCF Operations in Fleet Management > Passwords and show as "Connected".
• Please wait up to 60 minutes before running "Update Password" again.

Note: If you find that day 2 operations is still failing in Fleet Manager for the VCF Automation component even after updating and then remediating password, please ensure that the correct password alias is selected when retrying the task.  If required please create a new password in the locker with the new password.  

Additional Information

With the upcoming 9.x release, a new Fleet LCM / VCF Operations component will be introduced. The updated VCF Ops will support password rotation and also provide the ability to configure a policy for passwords to never expire out of the box.