Resolving 400 Bad Request Login Failures in VCF Operations Orchestrator 9.0.2.0 due to Secret Synchronization

Products

• VCF Automation

Issue/Introduction

Users may encounter "400 Bad Request" errors when attempting to log in to VCF Operations Orchestrator 9.0.2.0. This issue occurs during redirection to the Identity Broker, preventing access. This article explains the cause and the workaround for High Availability (HA) deployments.

Environment

• VCF Operations Orchestrator 9.0.2.0
• High Availability (HA) cluster deployment with multiple nodes.

Cause

In VCF Operations Orchestrator 9.0.2.0, a synchronization issue exists where multiple nodes simultaneously attempt to rotate the authentication secret. This concurrent rotation leads to data corruption, breaking the integration with the Identity Broker. 

Resolution

To resolve this issue, scale the VCF Operations Orchestrator appliance down to a single node. Use one of the following methods:

Method 1: Using kubectl

1. Log in to the management cluster.
2. Scale the orchestrator replicas to 1 using the following command: kubectl -n prelude scale deployment vco-app --replicas=1
3. After the fix is applied in a future release, restore the HA environment by scaling back up: kubectl -n prelude scale deployment vco-app --replicas=3

Method 2: Using vracli

1. Remove 2 nodes from the cluster using the following command: vracli cluster leave or vracli cluster remove
2. Execute the deploy.sh script to reconfigure the environment.

Note: The permanent fix is targeted for a future release. Users should subscribe to this article (reference: https://knowledge.broadcom.com/external/article/275360) to receive updates regarding the availability of the permanent code fix.

Additional Information

This is tracked under internal defect VCOPS-108509.