Domain user failed to schedule task at vCenter.

Products

• VMware vCenter Server

Issue/Introduction

When scheduling a task using domain user, the task failed to run with Reason of failure of “NotAuthenticated for the task operation.”

• In the /var/log/vmware/vpxd/vpxd.log file, the following error(s) are observed:

[yyyy-mm-ddThh:mm:ss] error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-##ID####] [AcquireToken] Failed to get token. Error:
--> Error:
-->    com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->
[yyyy-mm-ddThh:mm:ss] error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-##ID####] BadVapiConfiguration / NotAuthenticated Exception while running the task: Error:
-->    com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->

• In the /var/log/vmware/sso/tokenservice.log file, the following error is observed:

[yyyy-mm-ddThh:mm:ss] ERROR tokenservice[79:tomcat-http--41] [CorId=#######-#####-####-####-###########OpId=] [com.vmware.vcenter.tokenservice.vapi.TokenProviderImpl] Exchange failed due to invalid grant:
com.vmware.vcenter.tokenservice.exceptions.InvalidGrant: JWT token failed signature verification. Token:{"sub":"ExampleUser@domain","aud":"vmware-tes:vc:persistabletoken","act":{"sub":"vpxd-#######-#####-####-####-###########@vsphere.local"},"scope":"offline_access","iss":"https:\/\/domain.domain\/openidconnect\/domain_name","iat":##########,"jti":"########-####-####-####-###########"}

Cause

User failed to be authenticated because of stale token.

Resolution

1. Take a snapshot of the vCenter before running the script. If there are more than one vCenter in Enhanced Linked Mode, take powered-off snapshots of all the vCenter servers.
2. Open a SSH session to vCenter.
3. Switch to shell.

   shell

4. Enter VCDB.

   psql -U postgres  VCDB

5. Verify a token of target user exists. Check user id at the first column of the output.

   select * from vpx_sched_persistent_user_token;
   

6. Remove stale token from database. Replace <user-id> with the id from step5.

   delete from vpx_sched_persistent_user_token where usertoken_id=<user-id>;

7. Verify the token has been successfully removed.

   select * from vpx_sched_persistent_user_token;

8. Schedule a task at vCenter, verify if task could be run now.