Unable to Attach Tags to VMs in vCenter After Upgrading to 8.0

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server is part of ELM , and issue is limited to single vCenter.
• In vCenter Server > Menu > Tags & Custom Attributes under Tags, all tags are listed.
• When attempting to attach tags to a virtual machine , no tags are shown.
  
  
  
  
• Creating new tags in vCenter Server is successfully, but the newly created tags also do not appear when trying to attach them to a VM
• In the /var/log/vmware/vpxd-svc/vpxd-svcs.log 
  
  

  YYYY-MM-DD:T:HH:MM:SS [dataservice-1 [] ERROR com.vmware.cis.server.authentication.impl.TokenLoginContext  opId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx] Failed to get a renewable act-as HoK token
com.vmware.vim.sso.client.exception.InvalidTokenRequestException: Request is invalid: ns0:InvalidRequest: Access not authorized!
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenServiceImpl.java:1147) ~[wstClient.jar:?]

          at java.lang.Thread.run(Thread.java:750) [?:1.8.0_401]

  YYYY-MM-DD:T:HH:MM:SS [dataservice-7 [] ERROR com.vmware.cis.core.tagging.vmodl.MoTagManager  opId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx] Failed to query attached objects for given tag : ManagedObjectReference: type = InventoryServiceTag, value = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx, serverGuid = null
com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = cis.tagging.unauthenticated.downstream.error,
    defaultMessage = Failed to create session on down-stream service, please create a new session with a delegable token,
    args = <null>,
    params = <null>,
    localized = <null>

  
  
  

Environment

• vCenter Server 8.0

Cause

• This issue can be caused if Default users are missing under Administrator Group.
• Verify if all the Default users are listed in Administrators Group (Compare the output between working and non-working vCenters, and add any missing users)
  
  /usr/lib/vmware-vmafd/bin/dir-cli group list --name Administrators

Resolution

Note: Before making any changes, it is recommended to take offline snapshot of all vCenter servers in ELM and while restoring, restore snapshot of all  vCenter servers.

Proceed with the steps below

• List service accounts on vCenter server.
  
   /usr/lib/vmware-vmafd/bin/dir-cli svcaccount list
  
  
• Capture the SPS service account.
• Add the missing users to the Administrators group by using the commands below.
  
  /usr/lib/vmware-vmafd/bin/dir-cli group modify --name Administrators --add sps-xxxxx-xxx-xxx-xxx-xxxxxxx


• Restart vCenter services using the below command
  
  service-control --stop --all && service-control --start --all

 

Additional Information

• For more information on vCenter Permission refer the article  Using the "authz-doctor" tool to identify vCenter permission issues