Intermittent deployment failures with 404 errors "Workflow ID Not Found During Execution"

Products

• VCF Automation

Issue/Introduction

Service-to-service calls to orchestrator intermittently fail with HTTP 404 errors. Workflow execution requests return errors indicating that the specified workflow ID cannot be found, leading to extensibility execution failures (Error [10040]).

The issue is observed with the following symptoms:

• Workflow execution fails with error:"Workflow with id '<workflow-ID>' could not be found on endpoint 'https://embedded.orchestrator'"
• 404 NOT_FOUND response when accessing: /vco/api/workflows/{workflowId}/executions
• Error [10040] observed during extensibility execution
• Workflows appear unavailable or are unexpectedly filtered out

Environment

VCF Automation 9.x

Cause

This issue is caused by a mismatch in JWT token cache lookup within orchestrator.

When a request from a user without orchestrator roles reaches the orchestrator REST API, orchestrator resolves the user’s role using the JWT token and caches the result, indicating that the user has no roles. A subsequent request from VCF Automation uses a service token along with the same user token, which should elevate the user’s permissions. However, before applying this elevation, orchestrator checks the cached JWT token details.

Since the cache still reflects that the user has no roles, all workflows are filtered out for that user context. As a result, the requested workflow cannot be found, and orchestrator returns a 404 NOT_FOUND error.

Resolution

Broadcom is aware of this issue and a fix is planned for an upcoming release.

Workaround: 

Reduce the expiration time of the JWT token cache in orchestrator to minimize the duration of stale cache entries.

Steps:

1. Navigate to:
   Orchestrator → System Settings → Configuration Properties
2. Click on Add Property
3. Configure the following:
        Name: com.vmware.o11n.jwt-details-expiration-details
          Value: Set a lower value in seconds (for example, 60)
4. Save the configuration

Note:

• The default value is 1800 seconds (30 minutes).
• Reducing the cache expiration increases the number of calls to the authentication provider, as orchestrator will perform more frequent token validations.
• Start with a lower value such as 60 seconds and monitor the environment for any performance impact before making further adjustments.