vCenter status is Failed in the Component Configuration page

Products

• VCF Operations

Issue/Introduction

• Identity Broker component failed for vCenter
• In vCenter > Single Sign On > Configuration, the VCF SSO is not showing under Identity Sources and it is showing as "Embedded"
• The status of the vCenter component is failed when navigating in VCF Operations to  Fleet Management > Identity & Access > VCF Instances > Component Configuration

Environment

• VCF Operations 9.0.x
• VCF Identity Broker 9.0.x
• vCenter 9.0.0
• vCenter 9.0.1

Cause

The issue happens due to the trustmanagement credential expiring due to a known race condition in vCenter versions 9.0.0 and 9.0.1.

This can be confirmed by seeing the following error in the /var/log/vmware/trustmanagement/trustmanagement-svcs.log on the vCenter

ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl  opId=] SOAP faultcom.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: The account of the user trying to authenticate is locked. :: The account of the user trying to authenticate is locked. :: User account locked: {Name: trustmanagement-<guid> Domain: vsphere.local}

Resolution

Upgrade vCenter to version 9.0.2 where this issue is fixed.

To workaround the issue in vCenter 9.0.0 and 9.0.1:

1. Log in to the vCenter as root via SSH

2. Open the shell prompt

   shell

3. Backup the svcaccount_prestart_util.py file

   cp /usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py /root/svcaccount_prestart_util.py.backup

4. Modify the svcaccount_prestart_util.py file

   sed -i "s/AUTHZ_OPERATIONS_NOT_SUPPORTED_SVCACCOUNTS = \['sts', 'serviceaccountmgmt', 'vpxd-svcs-user', 'cis-license'\]/AUTHZ_OPERATIONS_NOT_SUPPORTED_SVCACCOUNTS = \['lookupsvc', 'trustmanagement', 'sts', 'serviceaccountmgmt', 'vpxd-svcs-user', 'cis-license'\]/" /usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py

5. Restart services on the vCenter (wait for command to complete before continuing to next step)

   service-control --stop --all && service-control --start --all

6. Log in to VCF Operations and navigate to Fleet Management > Identity & Access > VCF Instances > Component Configuration

7. Select the vCenter component and click Deregister Component

8. Select the vCenter component and click Configure Component

 

Note: If the workaround causes problems, the original svcaccount_prestart_util.py file can be restored and the services restarted

cp /root/svcaccount_prestart_util.py.backup /usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py

service-control --stop --all && service-control --start --all

Additional Information

vCenter 9.0.2 Release Notes