Scheduled VM Clone Task Fails with error "The token has expired"

Products

• VMware vCenter Server 8.0

Issue/Introduction

In a vSphere environment when a scheduled task configured to clone a Virtual Machine (VM) across vCenter Servers (ELM) or within same vCenter server the operation fails during execution.

When the scheduled task begins, the following symptoms are observed:

• The task fails with the error: "A general system error occurred: The token has expired at: YYYY-MM-DDT0HH:MM:SS".
  
  
• The vSphere Client (vsphere-ui) becomes unresponsive.
  
  
• The vpxd service crashes immediately upon task initiation.
  
  
• The /var/log/vmware/vpxd/vpxd.log file records the following failure during the vim.VirtualMachine.clone operation:
  
  

  YYYY-MM-DDT0HH:MM:SS info vpxd[2292927] [Originator@6876 sub=vpxLro opID=mi###sjq-351691-auto-####-h5:70###57-47] [VpxLRO] -- BEGIN task-7033 -- vm-#### -- vim.VirtualMachine.clone -- 520###60-####-ba11-####-7ca###d3894(52###e45-c##4-0##8-1##0-a66####8b8db)
  YYYY-MM-DDT0HH:MM:SS  info vpxd[2292927] [Originator@6876 sub=Vmprov opID=mi###sjq-351691-auto-####-h5:70###57-47] RetryHandler encountered unhandled Exception:N9SsoClient14ParseExceptionE(The token has expired at: YYYY-MM-DDT0HH:MM:SS)
  --> [context]zKq7AV#####AA7SfAEodnB4ZAAAyzd#####dm1hY29yZS5#####AArDA3AA9V####UiQJsaWJzc2#####llbnQuc28AAQG3########IgMBUC8DAZgw####gji7ZgKCQ5hlAoLTvGUCg############AKCb51cAoI0k1wCgsuTXAKCNJ################XAzbwCC2YKvAgRDbxxsaWJ2################CCsLiJAoLr+q0CgpsLrgKCNRmtAoJA9K0CAAT8LAAPVS0A6ypGBcSQCGxpYmMuc2[/context]
  
  YYYY-MM-DDT0HH:MM:SS warning vpxd[2292927] [Originator@6876 sub=Vmomi opID=mi###sjq-351691-auto-####-h5:70###57-47] VMOMI activation LRO failed;<<520###60-####-ba11-####-7ca###d3894, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 47988'>>, vm-####, vim.VirtualMachine.clone, (null),(null)>, N9SsoClient14ParseExceptionE(The token has expired at:YYYY-MM-DDT0HH:MM:SS
  )
  --> [context]zKq7AV#####AA7SfAEodnB4ZAAAyzd#####dm1hY29yZS5#####AArDA3AA9V####UiQJsa########gji7ZgKCQ5hlAoLTvGUCg############AKCb51cAoI0k1wCgsuTXAKCNJ################XAKCwrJcAoJsuhACgge7EAKD8Os5AWxpYnZpbS10eXBlcy5zbwCC2YKvAgRDbxxsaWJ2################CCsLiJAoLr+q0CgpsLrgKCNRmtAoJA9K0CAAT8LAAPVS0A6ypGBcSQCGxpYmMuc2[/context]
  --> 
  YYYY-MM-DDT0HH:MM:SS info vpxd[2292927] [Originator@6876 sub=vpxLro opID=mi###sjq-351691-auto-####-h5:70###57-47] [VpxLRO] -- FINISH task-7033
  YYYY-MM-DDT0HH:MM:SS  error vpxd[2292927] [Originator@6876 sub=Default opID=mi###sjq-351691-auto-####-h5:70###57-47] [VpxLRO] -- ERROR task-7033 -- 520###60-####-ba11-####-7ca###d3894(52###e45-c##4-0##8-1##0-a66####8b8db) -- vm-#### -- vim.VirtualMachine.clone: :vmodl.fault.SystemError
  --> Result:
  --> (vmodl.fault.SystemError) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    reason = "The token has expired at: YYYY-MM-DDT0HH:MM:SS"
  -->    msg = ""
  --> }
  --> Args:
  -->
  --> Arg folder:
  --> 'vim.Folder:group-####'
  --> Arg name:
  --> "<NAME_OF_VM>"
  --> Arg spec:
  --> (vim.vm.CloneSpec) {
  -->    location = (vim.vm.RelocateSpec) {
  -->       service = (vim.ServiceLocator) {
  -->          instanceUuid = "da###c3-####-4566-###-9e20####7df6",
  -->          url = "https://<FQDN_OF_vCENTER>:443/sdk",
  -->          credential = (vim.ServiceLocator.SAMLCredential) {
  -->             token = (not shown)
  -->          },
  -->          sslThumbprint = "3E:##:53:91:##:5D:3B:##:EB:A1:##:33:F5:##:2D:70:##:66:E6:45",
  -->          sslCertificate = "-----BEGIN CERTIFICATE-

  
  
  
  

 

Cause

When a scheduled task is created, the system is designed to "capture" the user's identity and session context to allow the task to run later without the user being logged in. However, in affected versions, the activation session locale is unset during this process.

When the scheduled interval triggers the task, the system attempts to reference the preserved identity context. Because the locale is missing, the system encounters a null pointer or an invalid session state, leading to an immediate "token expired" error and a subsequent crash of the vpxd service. This occurs for both local and Cross-VC (XVC) operations.

Resolution

Currently, there is no resolution.

Workaround: -

If the task has already been created, do not wait for the timer to trigger it. Instead, go to Scheduled Tasks view and select Run to execute it within an active, valid session.

Broadcom is aware of this issue, and a fix is planned for a future release of vCenter Server.