Scheduled workflows stop running in VCF Orchestrator 9
search cancel

Scheduled workflows stop running in VCF Orchestrator 9

book

Article ID: 443519

calendar_today

Updated On:

Products

VCF Automation

Issue/Introduction

  • Scheduled workflows were running for some time without issue, but stop running 
  • The account of the user who scheduled the workflows is locked.
  • After the AD account has been unlocked and the schedule is recreated, the workflows run again for a period of time and then stop again.
  • You do not see the issue when scheduling the workflows with a different account
  • Errors in the vco-app-server logs show the token renewal failing with a 401 similar to this:

    2026-05-19T00:35:12.029Z WARN vco [host='vco-app-##########-#####' thread='WorkflowExecutorPool-Thread-####' user='username@example.com' org='ORG_NAME' trace='-'] {|__SYSTEM|username@example.com:WORK_FLOW_NAME:########-####-####-####-############:token=########-####-####-####-############} ch.dunes.util.StubbornRetrier - Retry skipped because of non-retryable error: 401 Unauthorized: "{"errors":[{"code":"401","message":"Unauthorized"}]}"
    2026-05-19T00:35:12.029Z INFO vco [host='vco-app-##########-#####' thread='WorkflowExecutorPool-Thread-####' user='username@example.com' org='ORG_NAME' trace='-'] {|__SYSTEM|username@example.com:WORK_FLOW_NAME:########-####-####-####-############:token=########-####-####-####-############} com.vmware.iam.txaz.client.TxazClientImpl - Fetching the service token for clientId: ########-####-####-####-############
    2026-05-19T00:35:12.029Z INFO vco [host='vco-app-##########-#####' thread='WorkflowExecutorPool-Thread-####' user='username@example.com' org='ORG_NAME' trace='-'] {|__SYSTEM|username@example.com:WORK_FLOW_NAME:########-####-####-####-############:token=########-####-####-####-############} com.vmware.iam.txaz.client.VidbClient - Fetching the client credential grant token for client: ########-####-####-####-############
    2026-05-19T00:35:12.029Z INFO vco [host='vco-app-##########-#####' thread='WorkflowExecutorPool-Thread-####' user='username@example.com' org='ORG_NAME' trace='-'] {|__SYSTEM|username@example.com:WORK_FLOW_NAME:########-####-####-####-############:token=########-####-####-####-############} com.vmware.iam.txaz.client.VidbClient - Sending the request to https://<VIDB_SERVER@example.com>/acs/t/ORG_NAME/token
    2026-05-19T00:35:12.080Z WARN vco [host='vco-app-##########-#####' thread='WorkflowExecutorPool-Thread-####' user='username@example.com' org='ORG_NAME' trace='-'] {|__SYSTEM|username@example.com:WORK_FLOW_NAME:########-####-####-####-############:token=########-####-####-####-############} ch.dunes.util.StubbornRetrier - Retry skipped because of non-retryable error: 401 Unauthorized: "{"errors":[{"code":"401","message":"Unauthorized"}]}"

Environment

VCF Operations Orchestrator 9.x

Cause

AD user account has been locked out external to VCF Orchestrator

Resolution

  1. Recreate the schedule using a different account, such as a service account as per the documentation: Schedule  a Workflow
  2. Work with your Internal Active Directory team to determine and resolve the origin of the account lock outs. 
Powered by Wolken Software