vCenter login failure in VVF 9.1 - "Unable to login because you do not have permission on any vCenter Server systems connected ot this client"

Products

• VMware vCenter Server
• VMware vSphere Foundation

Issue/Introduction

• vCenter logins using SSO Admin account [email protected] fails in VVF 9.1 environment.
  
  
• Login fails with no permission error message as shown below:
  
  
  
  
• This issue is observed after 25 days of successful VVF 9.1 deployment or upgrade.
  
  
• Logs on vCenter shows the entries as below:
  
  

  /var/log/vmware/vpxd-svcs/authz-event.log

YYYY-MM-DDTHH:MM:SS.###Z INFO vpxd-svcs-authz-event ### [vc@#### threadName="authz-service-0" logger="AuthorizationService.AuditLog" opId="#####"] Action performed by principal(name=VSPHERE.LOCAL\serviceaccountmgmt-#######-####-####-####-#######,isGroup=false):Remove global access for [ Name=VSPHERE.LOCAL\Administrator,isGroup=false ]

  /var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDTHH:MM:SS.###Z INFO vpxd ##### [vc@#### sub="Authorize" opId="#####"] Skip event on permission removed by other service for User: VSPHERE.LOCAL\Administrator, Object: group-##
YYYY-MM-DDTHH:MM:SS.###Z INFO vpxd ##### [vc@#### sub="PermissionsImpl" opId="#####"] [Delete] Successfully deleted global permission: GlobalAcl:######-####-####-####-#####

Environment

vCenter 9.1

VMware vSphere Foundation 9.1

Cause

A software defect causes the LCM service to persist vCenter administrator (e.g. [email protected]) account rather than persisting a dedicated Service Account. LCM persisted Service Accounts undergo rotation every 25 days, and during this rotation workflow the LCM service attempts to delete the persisted vCenter administrator account. vCenter administrator account is a built-in default account and cannot be deleted, but the deletion attempt renders the account to become unusable for logins.

Resolution

Recreate the missing Administrator permissions by executing the attached script.

1. Download the attached script repair_admin_groups_and_perms.py.
2. Copy the script to vCenter using WinScp or copy paste the contents to a new file.
3. Modify the file permissions for script execution.
4. Execute the script as below:
   
   python repair_admin_groups_and_perms.py -p 'SSO Administrator Password'
   
   Sample Result:
   

   Reading credentials from lwregshell ...
     dcAccountDN : cn=vcenter.example.com,ou=Domain Controllers,dc=vsphere,dc=local
     domain DN   : dc=vsphere,dc=local
     DC name     : vcenter.example.com
     admin DN    : cn=Administrator,cn=Users,dc=vsphere,dc=local
   
     OK    cn=Users,cn=Builtin,dc=vsphere,dc=local
     OK    cn=Administrators,cn=Builtin,dc=vsphere,dc=local
     OK    cn=CAAdmins,cn=Builtin,dc=vsphere,dc=local
     OK    CN=SystemConfiguration.Administrators,dc=vsphere,dc=local
     OK    CN=SystemConfiguration.BashShellAdministrators,dc=vsphere,dc=local
     OK    CN=SystemConfiguration.ReadOnly,dc=vsphere,dc=local
     OK    CN=SystemConfiguration.SupportUsers,dc=vsphere,dc=local
   
   Groups: added=7  skipped=0  failed=0
   
   Scanning svcaccountmgmt.log for deleted Administrator permissions ...
     Found 2 deleted permission(s):
       Folder:#######-####-####-####-###########
       GlobalAcl:#######-####-####-####-###########
   
   Connecting to vCenter REST API as [email protected] ...
     OK    Inventory (group-d1)
     OK    Global ACL
     Permissions: restored=2  skipped=0  failed=0
   
   Done.

5. Retry login to vSphere Client using Administrator account.

Additional Information

To avoid this issue (if vCenter 9.1 installation or upgrade date < 25 days)

1. Change the password of SSO Administrator account eg. [email protected], refer Change Your vCenter Single Sign-On Password for more details.
2. Resetting the Administrator credential will lead to the vCenter credential persisted by LCM to get out of sync, and lead to Logs Management deployment workflow to fail until a dedicated vCenter Service Account is configured on LCM. Contact Global Support to update the password in LCM.