Scheduled VCF Operations Orchestrator workflows fails after about 1 hour.

Products

• VCF Operations

Issue/Introduction

Symptoms:

• Workflows failing randomly and in the logs the following error message can be seen: "Failed to retrieve permissions using service account".
• Users don't have the correct roles in the client. For example an Administrator does not see any workflows. And in the logs the same error message can be seen.
• The scheduled workflow is failing after an hour.

Environment

VCF Operations Orchestrator 9.0 External only
VCF Operations Orchestrator 9.1 External only

Cause

In 3-node HA there is a mis-synchronization between the nodes after rotating the secret for the orchestrator service account causing its secret to be saved in the database in invalid state.

Resolution

Workaround:

1. Re-register the authentication provider for VCFOperation Orchestrator via CLI so its secret gets corrected.
2. Login into VCFA in the Provider Management tenant (system tenant)
3. In the Right menu go to the "Access Control"
4. In the top menu go to "Service Accounts"
5. Find the service account for Orchestrator. It should start with ext-o11n- and have the correct hostname in the "Client URI" field. Start editing it
6. Disable the "Require Rotation" option and save it

Additional Information

Please subscribe to this KB article so you will be notified when the permanent fix will be available.