Subscription Workflows fail sporadically with "Delegating token is not service token" in VCF Automation Orchestrator 9.x

Products

• VCF Operations/Automation (formerly VMware Aria Suite)
• VCF Automation

Issue/Introduction

• When executing long-running workflows on a VMware Cloud Foundation Automation Orchestrator 9.x VM Apps Subscription, the workflow fails sporadically.

• The workflow failure message displays: "Delegating token is not service token."

• The Orchestrator logs may indicate a 401 Unauthorized error from the RBAC service when attempting to refresh the token data the workflow was started with.

Environment

VCF Automation Orchestrator 9.x

Cause

This issue occurs because the gateway service account tokens expire during the execution of long-running workflows. Orchestrator continues to check the gateway service account even after the token has expired, leading to the 401 Unauthorized response when attempting to refresh the data.

Resolution

This is a known issue affecting VCF Automation Orchestrator 9.x environments. Broadcom engineering is currently working on a permanent fix to be delivered in a future release.

Workaround

To resolve this issue temporarily, you can increase the accessTokenTimeToLive to 24 hours (86400 seconds) via the Cloud API. You can do this using a two-step manual process or a one-liner if you have jq installed.

Prerequisites:

• You will need the FQDN of your Tenant Manager (<tenant-manager-fqdn>).

• You must generate a System Admin Bearer Token (<your-system-admin-bearer-token>). Refer Generating API Tokens for Service Provider Account for steps

Option 1: Two-Step Method

Step 1: Retrieve the current OpenID Provider configuration

Run the following curl command to get the current configuration:

TM_URL="https://<tenant-manager-fqdn>"
TOKEN="<your-system-admin-bearer-token>"
curl -s -k GET \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/json;version=40.0"  \
  "$TM_URL/cloudapi/1.0.0/openIdProvider"

Step 2: Update the configuration with a 24-hour access token TTL

Take the activeKey object from the JSON response in Step 1 and paste it into the <activeKey-from-GET> placeholder below.

TM_URL="https://<tenant-manager-fqdn>"
TOKEN="<your-system-admin-bearer-token>"
curl -s -k -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json;version=40.0"  \
  "$TM_URL/cloudapi/1.0.0/openIdProvider" \
  -d '{
    "activeKey": <activeKey-from-GET>,
    "allowHttp": false,
    "authorizationCodeTimeToLive": 300,
    "idTokenTimeToLive": 3600,
    "accessTokenTimeToLive": 86400
  }'

Option 2: Automated One-Liner (Requires jq)

If you have jq installed, you can use this script to automatically read the current configuration, patch only the accessTokenTimeToLive value to 24 hours, and push the update in one execution:

 

TM_URL="https://<tenant-manager-fqdn>"
TOKEN="<your-system-admin-bearer-token>"

# Fetch current config
CURRENT=$(curl -s -k \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/json;version=40.0"  \
  "$TM_URL/cloudapi/1.0.0/openIdProvider")

# Update TTL using jq
UPDATED=$(echo "$CURRENT" | jq '.accessTokenTimeToLive = 86400')

# Push updated config
curl -s -k -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json;version=40.0"  \
  "$TM_URL/cloudapi/1.0.0/openIdProvider" \
  -d "$UPDATED"

 

Additional Information

We can also achieve this directly through the UI:

1. Log in to the Provider Portal using administrator credentials.

2. Navigate to Administration > OIDC Services > General Settings.

3. Click Edit next to Access Token Lifetime.

4. Update the value to 86400 seconds (24 hours) and save your changes.