Scheduled Snapshot Task Failing "NotAuthenticated" vCenter Server 8.0 (Persistent Token Issue)

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

Scheduled tasks failing to execute in vCenter Server environments, specifically snapshots, often return the error "NotAuthenticated for the task operation." This issue arises due to stale OAuth2 persistent tokens stored in the vCenter Database (VCDB) following certificate rotations or version upgrades, preventing successful task authentication.

• Scheduled tasks, such as snapshots, fail to trigger.
  
  
• Task execution fails with the error: "NotAuthenticated for the task operation."
  
  
• Relevant logs in /var/log/vmware/vpxd/vpxd.log display:
  [######] error vpxd[####] [Originator@#### sub=MoScheduledTask]
[AcquireToken] Failed to get token. Error:
--> Error:
--> com.vmware.oauth2.errors.invalid_grant
  
  
• Logs in /var/log/vmware/sso/tokenservice.log display:
  ERROR tokenservice... Exchange failed due to invalid grant: com.vmware.vcenter.tokenservice.exceptions.InvalidGrant: JWT token failed signature verification.

Environment

• vCenter server 8.0.2
• vCenter server 8.0.3 

Cause

• Persistent user tokens stored in the vpx_sched_persistent_user_token table in the VCDB become stale or invalid after certificate or version updates. These tokens fail signature verification, causing the Token Service to reject the task execution request.

Resolution

VMware by Broadcom Engineering is aware of the issue and is working on a permanent fix.

Workaround:

For affected user accounts, run the attached script to clear the stale token(s).
Note: Running this script clears the tokens for the specified users, requiring their scheduled tasks to be deleted and recreated.

1. Take a snapshot of the vCenter Server.
   If there are multiple vCenter Servers in Enhanced Linked Mode (ELM), take powered-off snapshots of all vCenter Servers in the SSO domain.
   
   
2. Download the remove_persistable_tokens.sh script to the vCenter Server /tmp directory.
   
   
3. Set the script permissions to executable by running the command: chmod +x /tmp/remove_persistable_tokens.sh
   
   
4. Identify the user account associated with the scheduled task initiator.
   
   
5. Execute the script using the following command format: ./remove_persistable_tokens.sh 'DOMAIN\User'  ( Note: The DOMAIN\User name is case sensitive.  It must match the exact syntax used in the scheduled task)
   
   
6. Navigate to the vSphere Client UI.
   
   
7. Delete the existing scheduled task.
   
   
8. Recreate the scheduled task. This action forces the vCenter Server to generate a new, valid persistent token for the task.

Additional Information

For environments managing large numbers of tasks, ensure that the vCenter Server is at the latest patch level to maintain optimal Token Service performance. If the issue persists after these steps, contact technical support to investigate potential database inconsistencies.

Troubleshooting Script Execution Errors:

         If the script fails with the following errors, it is likely due to hidden DOS carriage returns (\r) added to the script if it was downloaded or edited using a Windows-based text editor:

root@VC-XXX [ /tmp ]# ./remove_persistable_tokens.sh 'task-initiator1' : No such file or directory.sh: line 1: /etc/profile.d/vmware-vpostgres-config.sh ./remove_persistable_tokens.sh: line 3: $'\r': command not found ./remove_persistable_tokens.sh: line 10: syntax error near unexpected token `$'\r''

To resolve this, clean the script by running the sed command before executing it again:

sed -i -e 's/\r$//' /tmp/remove_persistable_tokens.sh

If the error persists after running the sed command, move the script to a different directory (e.g., /var/core), execute it from there, and ensure to delete the script after it runs successfully.

• vCenter Server のドメイン/管理者ユーザーのスケジュール設定タスクが「タスク操作について NotAuthenticated です」エラーで失敗する (446841)