Service Broker stuck in SYNCHRONIZING state during VMware Tanzu for Valkey upgrade

Products

• VMware Tanzu Platform Core
• VMware Tanzu for Valkey
• Redis for VMware Tanzu
• VMware Tanzu Redis
• VMware Tanzu for MySQL
• VMware Tanzu for Postgres

Issue/Introduction

During an upgrade or update of the VMware Tanzu for Valkey tile (formerly VMware Tanzu for Redis), users may encounter the following symptoms:

• Errand Failure: The broker-registrar errand fails with Exit Code 1.

• CLI Errors: Manual attempts to update the service broker via the Cloud Foundry CLI (cf update-service-broker) fail with the error:

  Cannot update a broker when other operation is already in progress

• Provisioning Failures: New service instance provisioning fails with a metadata mismatch error:

  provision could not be completed: Service broker error: passed maintenance_info does not match the catalog maintenance_info.

Environment

VMware Tanzu for Valkey

Cause

This issue occurs due to a persistent lock within the Cloud Controller Database (CCDB).

During a tile upgrade, the Cloud Controller (CC) synchronizes the new Valkey catalog. If this operation is interrupted, times out, or is blocked by other stalled service operations, the broker record becomes stuck in a SYNCHRONIZING state. To prevent metadata corruption, the Cloud Controller blocks all subsequent registration attempts while in this state.

This can also happen if there are any erroneous update to CCDB (CCNG #2937)

 

 

Resolution

Because resolving this issue requires direct modification of the Cloud Controller Database (CCDB), it is highly recommended to contact Broadcom Support for assistance. Please provide the deployment manifest and the logs from the failed broker-registrar errand when opening a case.