Domain/Admin user based Scheduled Tasks failing to run on vCenter with error: "NotAuthenticated for the task operation" when running Domain user Scheduled Tasks in vCenter 8.0"

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

Following an upgrade to vCenter 8.0, some environments may experience domain user created scheduled tasks failing to run.

• In the /var/log/vmware/vpxd/vpxd.log file, the following error(s) are observed:
  
  [yyyy-mm-ddThh:mm:ss] error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-##ID####] [AcquireToken] Failed to get token. Error:
--> Error:
-->    com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->
[yyyy-mm-ddThh:mm:ss] error vpxd[06802] [Originator@6876 sub=MoScheduledTask opID=ProcessScheduledTaskFiring-##ID####] BadVapiConfiguration / NotAuthenticated Exception while running the task: Error:
-->    com.vmware.oauth2.errors.invalid_grant
--> No messages!
-->
  
  
  
• In the /var/log/vmware/sso/tokenservice.log file, the following error is observed:
  
  [yyyy-mm-ddThh:mm:ss] ERROR tokenservice[79:tomcat-http--41] [CorId=#######-#####-####-####-###########OpId=] [com.vmware.vcenter.tokenservice.vapi.TokenProviderImpl] Exchange failed due to invalid grant:
com.vmware.vcenter.tokenservice.exceptions.InvalidGrant: JWT token failed signature verification. Token:{"sub":"ExampleUser@domain","aud":"vmware-tes:vc:persistabletoken","act":{"sub":"vpxd-#######-#####-####-####-###########@vsphere.local"},"scope":"offline_access","iss":"https:\/\/domain.domain\/openidconnect\/domain_name","iat":#######,"jti":"########-####-####-####-###########"}

Environment

• vCenter server 8.0.2
• vCenter server 8.0.3 

Cause

• Running the 'vCert' script does not currently remove existing persistable tokens from the vCenter database.
• This leads to scheduled tasks failing to run due to stale tokens.

Resolution

VMware by Broadcom Engineering is aware of the issue and is working on a permanent fix.

Workaround:

For affected user accounts, run the attached script to clear the stale token(s).
Note: Running this script clears the tokens for the specified users, requiring their scheduled tasks to be deleted and recreated.

1. Take a snapshot of the vCenter Server.
   If there are multiple vCenter Servers in Enhanced Linked Mode (ELM), take powered-off snapshots of all vCenter Servers in the SSO domain.
   
   
2. Download the attached script remove_persistable_tokens.sh and upload it to the /tmp directory of the vCenter Server.
   
   
3. Make the script executable:
   
   chmod +x remove_persistable_tokens.sh
   
   
4. Identify the Task Initiator (the user who created the task) from the Scheduled Task pane in the vSphere UI.
   
   
5. Run the script, passing the impacted Task Initiator(s) reported in the tokenservice.log as parameters.
   Note: Single quotation marks are required around the username.
   
   ./remove_persistable_tokens.sh 'DOMAIN\User1' 'DOMAIN\User2'
   
   (Example: ./remove_persistable_tokens.sh 'EXAMPLE\User')

6. Delete the existing scheduled task(s) in the vSphere UI and recreate them.

Additional Information

Troubleshooting Script Execution Errors:

         If the script fails with the following errors, it is likely due to hidden DOS carriage returns (\r) added to the script if it was downloaded or edited using a Windows-based text editor:

root@VC-XXX [ /tmp ]# ./remove_persistable_tokens.sh 'task-initiator1' : No such file or directory.sh: line 1: /etc/profile.d/vmware-vpostgres-config.sh ./remove_persistable_tokens.sh: line 3: $'\r': command not found ./remove_persistable_tokens.sh: line 10: syntax error near unexpected token `$'\r''

To resolve this, clean the script by running the sed command before executing it again:

sed -i -e 's/\r$//' /tmp/remove_persistable_tokens.sh

If the error persists after running the sed command, move the script to a different directory (e.g., /var/core), execute it from there, and ensure to delete the script after it runs successfully.

• vCenter Server のドメイン/管理者ユーザーのスケジュール設定タスクが「タスク操作について NotAuthenticated です」エラーで失敗する (446841)