Automated Orchestrator policies fail to trigger after upgrade to VCF Automation 9
search cancel

Automated Orchestrator policies fail to trigger after upgrade to VCF Automation 9

book

Article ID: 446432

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • Automated Orchestrator policies do not trigger automatically after upgrade
  • Policy execution works as expected when manually triggered
  • Newly created or duplicated policies in VCF Automation 9.x function correctly
  • Orchestrator logs show
    2026-05-26T10:50:00.751Z WARN vco [host='vco-app-1' thread='CustomThreadPool's Thread-1' user='' org='' trace=''] {} com.vmware.o11n.security.csp.rbac.RbacClient - Received 401 from RBAC. Access token expired/invalidated? Attempting to retrieve permissions of null using Orchestrator service account
2026-05-26T10:50:00.751Z ERROR vco [host='vco-app-1' thread='CustomThreadPool's Thread-1' user='' org='' trace=''] {} com.vmware.o11n.security.csp.CspUtil - Could not verify if token is service account
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Empty roles context.
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2086) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.cache.LocalCache.get(LocalCache.java:4012) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4922) ~[guava-32.1.2-jre.jar:?]
        at com.vmware.o11n.security.csp.CspUtil.isServiceAccount(CspUtil.java:719) [o11n-security-csp-provider-9.0.2.0.jar:?]
        at com.vmware.o11n.security.csp.web.VcoCspAuthenticationConverter.convert(VcoCspAuthenticationConverter.java:121) [o11n-security-csp-provider-9.0.2.0.jar:?]
        at com.vmware.o11n.security.csp.web.VcoCspAuthenticationConverter.convert(VcoCspAuthenticationConverter.java:115) [o11n-security-csp-provider-9.0.2.0.jar:?]
        at com.vmware.o11n.security.csp.VcoCspAuthenticationProvider.authenticate(VcoCspAuthenticationProvider.java:120) [o11n-security-csp-provider-9.0.2.0.jar:?]
        at com.vmware.o11n.security.auth.impl.AuthenticationProviderManager.performAuthenticationOnProvider(AuthenticationProviderManager.java:131) [o11n-security-core-9.0.2.0.jar:?]
        at com.vmware.o11n.security.auth.impl.AuthenticationProviderManager.authenticate(AuthenticationProviderManager.java:111) [o11n-security-core-9.0.2.0.jar:?]
        at com.vmware.o11n.security.auth.impl.AuthenticationProviderManager.authenticate(AuthenticationProviderManager.java:100) [o11n-security-core-9.0.2.0.jar:?]
        at com.vmware.o11n.security.session.SessionManager.loadTokenForSession(SessionManager.java:279) [o11n-security-core-9.0.2.0.jar:?]
        ...

2026-05-26T10:50:00.752Z WARN vco [host='vco-app-1' thread='CustomThreadPool's Thread-1' user='' org='' trace=''] {} com.vmware.o11n.service.tasks.TaskRunner - Unable to execute workflow from task
com.vmware.o11n.security.csp.NoOrgIdException: The 'context_name' (orgId) must not be 'null' or empty in jwt token.
        at com.vmware.o11n.security.csp.web.VcoCspAuthenticationConverter.convert(VcoCspAuthenticationConverter.java:141) ~[o11n-security-csp-provider-9.0.2.0.jar:?]
        at com.vmware.o11n.security.csp.web.VcoCspAuthenticationConverter.convert(VcoCspAuthenticationConverter.java:115) ~[o11n-security-csp-provider-9.0.2.0.jar:?]
        at com.vmware.o11n.security.csp.VcoCspAuthenticationProvider.authenticate(VcoCspAuthenticationProvider.java:120) ~[o11n-security-csp-provider-9.0.2.0.jar:?]
        at com.vmware.o11n.security.auth.impl.AuthenticationProviderManager.performAuthenticationOnProvider(AuthenticationProviderManager.java:131) ~[o11n-security-core-9.0.2.0.jar:?]
        at com.vmware.o11n.security.auth.impl.AuthenticationProviderManager.authenticate(AuthenticationProviderManager.java:111) ~[o11n-security-core-9.0.2.0.jar:?]
        at com.vmware.o11n.security.auth.impl.AuthenticationProviderManager.authenticate(AuthenticationProviderManager.java:100) ~[o11n-security-core-9.0.2.0.jar:?]
        at com.vmware.o11n.security.session.SessionManager.loadTokenForSession(SessionManager.java:279) ~[o11n-security-core-9.0.2.0.jar:?]
        ...

2026-05-26T10:50:00.753Z WARN vco [host='vco-app-1' thread='CustomThreadPool's Thread-1' user='' org='' trace=''] {} com.vmware.o11n.service.tasks.RunnableTask - Exception while executing task: EXAMPLE
2026-05-26T10:50:00.753Z WARN vco [host='vco-app-1' thread='CustomThreadPool's Thread-1' user='' org='' trace=''] {} com.vmware.o11n.service.tasks.RunnableTask - ==== Task 'EXAMPLE' failed

Environment

VCF Operations Orchestrator 9.x

Cause

The issue is typically observed with policies that were created in earlier versions Aria Automation Orchestrator 8 and carried over through the upgrade to version VCF Operations Orchestrator 9.

The underlying mechanism for task scheduling or the security context (RBAC) associated with these legacy policies may fail to validate correctly in the new version, leading to execution failures.

Resolution

To resolve this issue, the affected policies must be recreated or duplicated within the VCF Automation 9.x environment to ensure they are registered correctly with the updated policy engine and RBAC provider.

  1. Log in to the VCF Automation organisation.
  2. Navigate to Orchestrator Service > Library > Policies.
  3. Identify the policy that is failing to run on schedule.
  4. Option A: Duplicate the Policy
    • Select the existing policy and use the Duplicate action.
    • Provide a new name and save the duplicated policy.
    • Enable the duplicated policy and verify the schedule.
  5. Option B: Create a New Policy
    • Create a new policy from scratch using the same workflow and configuration as the original.
    • Apply the policy to the required objects.
  6. Once the new or duplicated policy is verified to be working, Disable or Delete the original legacy policy.

Workaround

If the policy engine continues to experience issues, users can transition to using Schedules directly within the Orchestrator service. Schedules in version 9.x have the ability to run as a System User, which may offer better stability than legacy policies.

Powered by Wolken Software