Domain user based Scheduled Tasks failing to run on vCenter with error "NotAuthenticated for the task operation"

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

Following an upgrade to vCenter 8.0, some environments may experience domain user created scheduled tasks failing to run.

• In the /var/log/vmware/vpxd/vpxd.log file, the following error(s) are observed:

2024-11-12T11:07:02.475+01:00 error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-********] [AcquireToken] Failed to get token. Error:
--> Error:
-->    com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->
2024-11-12T11:07:02.478+01:00 error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-********] BadVapiConfiguration / NotAuthenticated Exception while running the task: Error:
-->    com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->

• In the /var/log/vmware/sso/tokenservice.log file, the following error is observed:

2024-11-12T10:07:02.468Z ERROR tokenservice[79:tomcat-http--41] [CorId=********-****-****-****-************ OpId=] [com.vmware.vcenter.tokenservice.vapi.TokenProviderImpl] Exchange failed due to invalid grant:
com.vmware.vcenter.tokenservice.exceptions.InvalidGrant: JWT token failed signature verification. Token:{"sub":"ExampleUser@domain","aud":"vmware-tes:vc:persistabletoken","act":{"sub":"vpxd-*******-****-****-****-************@vsphere.local"},"scope":"offline_access","iss":"https:\/\/domain.domain\/openidconnect\/domain_name","iat":1722332383,"jti":"********-****-****-****-************"}

Environment

vCenter server 8.0.2

vCenter server 8.0.3 

Cause

• Running the 'fixsts' script does not currently remove existing persistable tokens from the vCenter database.
• This leads to scheduled tasks failing to run due to stale tokens.

Resolution

• Engineering is aware of the issue and is working on a permanent fix.

Workaround:

For affected users accounts, a script can be run to clear the stale token(s):

1. Take a snapshot of the vCenter before running the script. If there are more than one vCenter in Enhanced Linked Mode, take powered-off snapshots of all the vCenter servers
2. Download the attached script 'remove_persistable_tokens.sh' and upload it to the /tmp directory of the vCenter.
3. Make the script as executable by running the command,

chmod +x remove_persistable_tokens.sh

     4. Run the script with the impacted usernames which was reported in the tokenservice.log as the parameters. This will cause all the previously scheduled tasks for the users to fail

./remove_persistable_tokens.sh 'your.domain\test1' 'your.domain\test2' ...

     5. Remove the existing scheduled task and re-schedule again

Additional Information

• Following error is observed, if the script is downloaded and uploaded to /tmp directory:

root@VC-XXX [ /tmp ]#./remove_persistable_tokens.sh 'your.domain\test1' 'your.domain\test2' ..."
: No such file or directory.sh: line 1: /etc/profile.d/vmware-vpostgres-config.sh
: No such file or directory.sh: line 2: /etc/profile.d/VMware-visl-integration.sh
./remove_persistable_tokens.sh: line 3: $'\r': command not found
./remove_persistable_tokens.sh: line 9: $'\r': command not found
./remove_persistable_tokens.sh: line 10: syntax error near unexpected token `$'do\r''
'/remove_persistable_tokens.sh: line 10: `for param in "$@"; do

• Run the following command: 

  sed -i -e 's/\r$//' remove_persistable_tokens.sh

• Now, execute the command with the impacted usernames 

./remove_persistable_tokens.sh 'your.domain\test1' 'your.domain\test2'