Password rotation workflow fails after renaming a Vmware Cloud Foundation management VM, PSC or vCenter

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Password Rotation workflow fails to rotate passwords in SDDC Manager UI. 
• The PSCs and/or vCenter friendly names have been changed in the vSphere Web Client inventory to something other than what was configured during VMware Cloud Foundation installation.
• You see messages similar to the following in the /home/vrack/vrm/logs/vrack-vrm-debug.log file on the SDDC Manager Controller VM.

com.vmware.vrack.vrm.cms.api.Changer$Fault: Expect 1 but 0 found for VirtualMachine:psc-1

Caused by: com.vmware.vrack.vrm.cms.api.Changer$Fault: Expect 1 but 0 found for VirtualMachine:psc-1
Caused by: com.vmware.vrack.vrm.cms.api.Changer$Fault: Expect 1 but 0 found for VirtualMachine:psc-1

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment

 

 

Environment

VMware Cloud Foundation 2.x

Cause

Renaming VMware Cloud Foundation management VMs can prevent SDDC Manager automation from running workflows. 

Resolution

1. SSH to the SDDC Manager Controller VM.
2. Remove failed password workflows by running the following commands:

/home/vrack/bin/list-password-workflows

Note: Make a note of the ID value for the failed password workflow

/home/vrack/bin/delete-password-workflow --id <ID value from previous command>

2. In the vSphere Web Client, rename any management VMS (PSCs, vCenter Servers, NSX Managers, Log Insight VMS) that have been renamed back to their default names.
3. In the vSphere Web Client, navigate to Home > Administration > Deployments > System Configuration > Nodes. 
4. Select each PSC or vCenter node then click on the Manage tab > Settings > Networking and verify that each node has the correct FQDN and DNS info on the network tab. 

Note:  If any show host (null) or are blank, note it and disregard.  The next step should remediate the issue and then the correct info should be shown.

5. Highlight each vCenter and PSC and restart the Appliance Management service under Related Items. 

​Note: Wait a few minutes after service restart. If any PSC or vCenter still does not show the correct DNS info and FQDN, manually verify whether the node in question has the correct FQDN, DNS servers, DNS records and IP address via ssh/console to the node or by accessing the node's vami web page. Remediate where needed and restart the Appliance Management service again. 

6. Issue the following command to obtain the root password for the management VMs:

/home/vrack/bin/lookup-passwords

7. Validate the passwords returned by attempting to access each of the management VMs. Remediate any password or expiry issues as needed.
8. SSH to SDDC Manager Controller VM and run /opt/vmware/sddc-support/sos --health-check to verify the environment is in a state healthy enough to run SDDC Manager workflows from the UI.  Remediate any "RED' items that could interfere with the workflow.
9. Return to SDDC manager and run Password Rotate workflow.