Following an upgrade to vCenter 8.0, some environments may experience domain user created scheduled tasks failing to run.
/var/log/vmware/vpxd/vpxd.log
file, the following error(s) are observed:[yyyy-mm-ddThh:mm:ss] error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-##ID####] [AcquireToken] Failed to get token. Error:
--> Error:
--> com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->
[yyyy-mm-ddThh:mm:ss] error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-##ID####] BadVapiConfiguration / NotAuthenticated Exception while running the task: Error:
--> com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->
/var/log/vmware/sso/tokenservice.log
file, the following error is observed:[yyyy-mm-ddThh:mm:ss] ERROR tokenservice[79:tomcat-http--41] [CorId=#######-#####-####-####-###########OpId=] [com.vmware.vcenter.tokenservice.vapi.TokenProviderImpl] Exchange failed due to invalid grant:
com.vmware.vcenter.tokenservice.exceptions.InvalidGrant: JWT token failed signature verification. Token:{"sub":"ExampleUser@domain","aud":"vmware-tes:vc:persistabletoken","act":{"sub":"vpxd-#######-#####-####-####-###########@vsphere.local"},"scope":"offline_access","iss":"https:\/\/domain.domain\/openidconnect\/domain_name","iat":1722332383,"jti":"########-####-####-####-###########"}
Workaround:
For affected users accounts, a script can be run to clear the stale token(s):
chmod +x remove_persistable_tokens.sh
4. Find out the task-initiator's name from the Scheduled Task pane in the UI. Replace "username" with "task-initiator"
5. Run the script with the impacted Task Initiators which was reported in the tokenservice.log as the parameters. This will cause all the previously scheduled tasks for the users to fail
./remove_persistable_tokens.sh 'task-initiator1' task-initiator2' ...
6. Remove the existing scheduled task and re-schedule again
root@VC-XXX [ /tmp ]#./remove_persistable_tokens.sh 'task-initiator1' 'task-initiator2'
: No such file or directory.sh: line 1: /etc/profile.d/vmware-vpostgres-config.sh
: No such file or directory.sh: line 2: /etc/profile.d/VMware-visl-integration.sh
./remove_persistable_tokens.sh: line 3: $'': command not found
./remove_persistable_tokens.sh: line 9: $'': command not found
./remove_persistable_tokens.sh: line 10: syntax error near unexpected token `$'do''
'/remove_persistable_tokens.sh: line 10: `for param in "$@"; do
sed -i -e 's/$//' remove_persistable_tokens.sh
./remove_persistable_tokens.sh 'task-initiator1' 'task-initiator2'