Identity provider fetch and tag assignment failures in vCenter 9.0

Products

• VMware vCenter Server

Issue/Introduction

Administrators may encounter failures when attempting to log in to vCenter Server or manage object tags. This issue typically presents with an error regarding identity provider fetching or an internal server error during tag operations. These symptoms often appear 90 days after the deployment or the last service restart.

• Login attempts to vCenter Server using domain credentials fail with the UI error: 
  
  An error occurred while fetching identity providers. Try again later. If the problem persists, contact your administrator.
  
  
• Tag assignment, creation, or category management via the vSphere Client UI or API fails with an Internal Server Error.
• The Inventory Service (invsvc) binaries and configuration directories /usr/lib/vmware-invsvc and /etc/vmware-invsvc may be missing or the service is not listed in service-control --status.
• The vmware-trustmanagement service is in a Stopped state.
• In /var/log/vmware/sso/websso.log, the following exception occurs: 
  
  com.vmware.identity.samlservice.ExtIdpNotFoundException: Exception while processing External login request : No External IDP configured.
  
  
• Log analysis shows internal server errors related to privilege checks: com.vmware.cis.authz/privilege.

Environment

• vCenter 9.0.0
• vCenter 9.0.1

Cause

This issue is caused by a race condition during the automated 90-day internal service account rotation. The utility script svcaccount_prestart_util.py incorrectly handles dependencies for the trustmanagement, sts, and lookupsvc services, preventing the vmware-trustmanagement service from starting. In vCenter 9.0, the legacy Inventory Service (invsvc) has been integrated into this new trust management architecture.

Resolution

This issue is resolved in vCenter 9.0.2. Log in to the Broadcom Support Portal to download this patch, depending on your entitlement, VMware vSphere Foundation or VMware Cloud Foundation.

1. To prevent the race condition from happening in vCenter versions 9.0.0 and 9.0.1, perform the following commands:
   

   cp /usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py /root/svcaccount_prestart_util.py.backup

   sed -i "s/AUTHZ_OPERATIONS_NOT_SUPPORTED_SVCACCOUNTS = \['sts', 'serviceaccountmgmt', 'vpxd-svcs-user', 'cis-license'\]/AUTHZ_OPERATIONS_NOT_SUPPORTED_SVCACCOUNTS = \['lookupsvc', 'trustmanagement', 'sts', 'serviceaccountmgmt', 'vpxd-svcs-user', 'cis-license'\]/" /usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py

   service-control --stop --all && service-control --start --all

2. If service fail to operate as expected, to roll back this change by performing the following commands:
   

   cp /root/svcaccount_prestart_util.py.backup /usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py

   service-control --stop --all && service-control --start --all

Additional Information

This can occur 90 days after the vCenter is deployed. The issue comes from a race condition affecting trustmanagement, sts, and lookupsvc services during an occurrence of internal service account rotations.