Scheduling Snapshot Fails With Error: "A general system error occurred: Error while getting Persistable Token for Session User from TES"

Products

• VMware vCenter Server

Issue/Introduction

• The following error may be observed when attempting to create a scheduled task, such as taking a snapshot or performing other actions in the vCenter Server.

  A general system error occurred: Error while getting Persistable Token for Session User from TES.

   

• Error in /var/log/vmware/vpxd/vpxd.log

  YYYY-MM-DD HH:MM:SS info vpxd[05874] [Originator@6876 sub=vpxLro opID=####-####-####-####] [VpxLRO] -- BEGIN task-######-- ScheduledTaskManager -- vim.scheduler.ScheduledTaskManager.create -- ####-####-####-####(####-####-####-####)
  
  YYYY-MM-DD HH:MM:SS info vpxd[05874] [Originator@6876 sub=SsoClient opID=####-####-####-####] Successfully acquired token: SamlToken [subject={Name: vpxd-####-####-####-####; Domain:vsphere.local}, groups=[{Name: Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], ... ]
  
  YYYY-MM-DD HH:MM:SS error vpxd[05874] [Originator@6876 sub=MoScheduledTask opID=####-####-####-####] Failed to get persistable Token: Unexpected SOAP fault: ns0:InvalidRequest; request failed.
  
  YYYY-MM-DD HH:MM:SS error vpxd[05874] [Originator@6876 sub=Default opID=####-####-####-####] [VpxLRO] -- ERROR task-######-- ####-####-####-####(####-####-####-####) -- ScheduledTaskManager -- vim.scheduler.ScheduledTaskManager.create: :vmodl.fault.SystemError
  --> Result:
  --> (vmodl.fault.SystemError) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    reason = "Error while getting Persistable Token for Session User from TES"
  }
  

Environment

VMware vCenter Server 8.x

Cause

• This issue occurs when the VPXD solution user is not a member of the ActAsUsers group within the vsphere.local Single Sign-On (SSO) domain. Membership in this group is a prerequisite for retrieving persistable session tokens, which are required for executing scheduled tasks and certain other vCenter operations.
• VPXD solution user (vpxd-<machine-id>) was missing from the ActAsUsers group. As a result, vCenter is unable to retrieve the necessary token for the session user, leading to failure in scheduling tasks.
• These log entries confirm that the failure occurs due to the system being unable to generate a persistable token for the VPXD solution user—an operation that requires ActAsUsers group membership.

Resolution

⚠️ Important: Before proceeding, take a full backup or offline snapshot of the vCenter Server(s). If using Enhanced Linked Mode, ensure all linked vCenter servers are backed up. Refer KB VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

 

Follow the steps below to verify and resolve the issue:

1. Access the vCenter Shell
   1. SSH into the vCenter Server with root user
   2. Enable the shell by running:

      shell

2. Retrieve the Machine ID
   • Run the following command to obtain the machine ID of your vCenter:

     /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

     Sample output

     root@vcsa01 [ ~ ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
     a367####-####-####-####-########fbba

     Note the output <machine-id> for use in subsequent steps.
     
     
3. Verify VPXD Solution User Exists
   • Check whether the vpxd-<machine-id> solution user exists:

     ldapsearch -o ldif-wrap=no -LLL -h localhost -b "dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w '<SSOdminPassword>' | grep "sAMAccountName: vpxd-<machine-id>"

     Sample

     ldapsearch -o ldif-wrap=no -LLL -h localhost -b "dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w '#########' | grep "sAMAccountName: vpxd-a367####-####-####-####-########fbba"

     • If the user exists, you will see:

       sAMAccountName: vpxd-a367####-####-####-####-########fbba

     • If it does not exist, create the user using:

       /vmafd/bin/dir-cli svcaccount create --name vpxd-<machine-id>

4. Check Members of ActAsUsers Group
   • Run the following command to list current members of the ActAsUsers group:

     ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=ActAsUsers,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w '<SSOdminPassword>' member

     You may see entries like:

     CN=machine-<machine-id>,CN=ServicePrincipals,DC=vsphere,DC=local CN=vsphere-webclient-<machine-id>,CN=ServicePrincipals,DC=vsphere,DC=local CN=vpxd-svc-<machine-id>,CN=ServicePrincipals,DC=vsphere,DC=local

     If vpxd-<machine-id> is not listed, continue to the next step.
     
     
5. Add VPXD User to ActAsUsers Group
   • Add the missing solution user using the following command:

     /usr/lib/vmware-vmafd/bin/dir-cli group modify --name ActAsUsers --add vpxd-<machine-id>

6. Verify Resolution
   • Once the user is added, retry creating the scheduled task in vCenter. The operation should now complete without the error.