Domain/Admin user based Scheduled Tasks failing to run on vCenter with error "NotAuthenticated for the task operation"

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

Following an upgrade to vCenter 8.0, some environments may experience domain user created scheduled tasks failing to run.

• In the /var/log/vmware/vpxd/vpxd.log file, the following error(s) are observed:

[yyyy-mm-ddThh:mm:ss] error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-##ID####] [AcquireToken] Failed to get token. Error:
--> Error:
-->    com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->
[yyyy-mm-ddThh:mm:ss] error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-##ID####] BadVapiConfiguration / NotAuthenticated Exception while running the task: Error:
-->    com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->

• In the /var/log/vmware/sso/tokenservice.log file, the following error is observed:

[yyyy-mm-ddThh:mm:ss] ERROR tokenservice[79:tomcat-http--41] [CorId=#######-#####-####-####-###########OpId=] [com.vmware.vcenter.tokenservice.vapi.TokenProviderImpl] Exchange failed due to invalid grant:
com.vmware.vcenter.tokenservice.exceptions.InvalidGrant: JWT token failed signature verification. Token:{"sub":"ExampleUser@domain","aud":"vmware-tes:vc:persistabletoken","act":{"sub":"vpxd-#######-#####-####-####-###########@vsphere.local"},"scope":"offline_access","iss":"https:\/\/domain.domain\/openidconnect\/domain_name","iat":1722332383,"jti":"########-####-####-####-###########"}

Environment

• vCenter server 8.0.2
• vCenter server 8.0.3 

Cause

• Running the 'fixsts' script does not currently remove existing persistable tokens from the vCenter database.
• This leads to scheduled tasks failing to run due to stale tokens.

Resolution

VMware by Broadcom Engineering is aware of the issue and is working on a permanent fix.

Workaround:

For affected users accounts, a script can be run to clear the stale token(s):

1. Take a snapshot of the vCenter before running the script. If there are more than one vCenter in Enhanced Linked Mode, take powered-off snapshots of all the vCenter servers
2. Download the attached script 'remove_persistable_tokens.sh' and upload it to the /tmp directory of the vCenter.
3. Make the script as executable by running the command:

   chmod +x remove_persistable_tokens.sh
   

   
   
   
4. Find out the task-initiator's name from the Scheduled Task pane in the UI. Replace "username" with "task-initiator"
5. Run the script with the impacted Task Initiators which was reported in the tokenservice.log as the parameters. This will cause all the previously scheduled tasks for the users to fail
   

   ./remove_persistable_tokens.sh 'task-initiator1' task-initiator2' ...
   
   eg. ./remove_persistable_tokens.sh 'EXAMPLE\User'

   
   
   
6. Execute the below command if script is failing with error "./remove_persistable_tokens.sh: line 3: $'': command not found". This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor.
   
   

   sed -i -e 's/\r$//' remove_persistable_tokens.sh
   

7. Remove the existing scheduled task and re-schedule again

Additional Information

• Following error is observed, if the script is downloaded and uploaded to /tmp directory:

root@VC-XXX [ /tmp ]#./remove_persistable_tokens.sh 'task-initiator1' 'task-initiator2'
: No such file or directory.sh: line 1: /etc/profile.d/vmware-vpostgres-config.sh
: No such file or directory.sh: line 2: /etc/profile.d/VMware-visl-integration.sh
./remove_persistable_tokens.sh: line 3: $'': command not found
./remove_persistable_tokens.sh: line 9: $'': command not found
./remove_persistable_tokens.sh: line 10: syntax error near unexpected token `$'do''
'/remove_persistable_tokens.sh: line 10: `for param in "$@"; do

• Run the following command: 

  sed -i -e 's/\r$//' remove_persistable_tokens.sh

• Now, execute the command with the impacted task initiators:

./remove_persistable_tokens.sh 'task-initiator1' 'task-initiator2' 

• If the above error is still observed, move the script to a different directory in the vCenter (Eg: /var/core) and ensure to remove the script after successfully running it.