Password update fails or accounts are shown as disconnected with message 'Failed to find powered-on VM with FQDN' or 'Found <X> powered-on VMs with FQDN, should be 1'

Products

• VMware Cloud Foundation
• VMware vCenter Server
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Password update in SDDC Manager or VCF Operations fails with message 'Failed to find powered-on VM with FQDN' or 'Found <X> powered-on VMs with FQDN, should be 1'.
• Accounts are shown as disconnected on Password Management UI and /var/log/vmware/vcf/operationsmanager/operationsmanager.log contains one of above two error messages.
• VCF may have recently been upgraded

Environment

• VMware Cloud Foundation
• vCenter Server 
• SDDC Manager

Cause

This issue can occur for VxRail Manager, vCenter or backup accounts when either of the following is true:

1. DNS name of a Virtual Machine selected from inventory of vCenter UI is different than the FQDN shown on Password Management UI.
2. Virtual Machine on which password update was tried is powered-off.
3. There are multiple Virtual Machine's with same DNS name in vCenter.
4. SDDC Manager is upgraded while VxRail Manager is yet to be upgraded as per the BOM.
   Here, DNS name on VxRail Manager Virtual Machine might not be set to the correct value during deployment which is a known issue.

Resolution

This issue is about configuration mismatch and is expected to be resolved from the user's end.

Workaround:

Identify if issue falls into cause a, b, c or d. Follow the steps mentioned below to resolve and retry password update.

Step to resolve -

• For cause a and d, update DNS name for the Virtual Machine in vCenter to be same as the value shown in Password Management UI. Follow the below mentioned steps to update the DNS name:

1. Navigate to the Virtual Machine in inventory from vCenter UI.
2. Open web console
3. Login with root user credentials into respective Virtual Machine(vCenter, VxRail Manager or SDDC-Manager)
4. Run following command: hostname <vm-fqdn-from-password-management-page>
5. Verify the 'DNS Name' to be updated for the Virtual Machine when selected from vCenter inventory after some time.

e.g. If DNS Name of a Virtual Machine is shown as vcenter-1 on vCenter but Password Management page shows it as vcenter-1.vsphere.local Virtual Machine then run command 'hostname vcenter-1.vsphere.local' on vCenter after logging in as root.

• For cause b, power on the Virtual Machine.
• For cause c, Password Management expects the DNS name on the powered-on Virtual Machines to be unique so only 1 powered-on Virtual Machine should have the DNS name value same as the FQDN shown on Password Management page. Either power-off the other Virtual Machines or change the DNS name using the resolution steps mentioned for cause a. This issue is likely to occur when a cold-clone of the vCenter Server VM it taken prior to an upgrade and the cloned VM is accidentally powered on at a later time.