VCF Identity Broker upgrade precheck (9.1.0 to 9.1.x) fails with "[VCFMS-PRECHECK-GENERIC-001]" and "Failed to list Spilo leader pods"

Products

• VCF Operations

Issue/Introduction

• When attempting to upgrade the VCF Identity Broker from version 9.1.0 to 9.1.x on the VCF Services runtime, the operation fails during the Run Identity broker upgrade prechecks stage.
  
• As seen in the VCF Operations UI, the subtask Database Upgrade Precheck fails with the following generic error message:

  "An unknown error occurred. Please retry the operation or contact support if the issue persists. [VCFMS-PRECHECK-GENERIC-001]"

  

• Additionally, checking the pods on VCF Services Runtime Control Plane in the vmsp-platform namespace reveals Argo workflow execute-script pods stuck in an Error state:

  #kubectl get pods -n vmsp-platform --sort-by=.metadata.creationTimestamp | grep -E "upgrade9x-postgres-precheck|upgrade-precheck-component" | grep Error
  
  NAMESPACE        NAME                                                                 READY   STATUS    RESTARTS   AGE
  vmsp-platform    upgrade9x-postgres-precheck-#####-execute-script-##########          0/2     Error     0          #m##s
  vmsp-platform    upgrade-precheck-component-#########-submit-workflow-##########      0/1     Error     0          #m##s

• Checking the logs on VCF Services Runtime Control Plane of the failed upgrade9x-postgres-precheck pod reveals the exact point of failure:

  #kubectl logs upgrade9x-postgres-precheck-#####-execute-script-########## -n vmsp-platform | less
  
  INFO : Blue/green upgrade detected
  ERROR : Failed to list Spilo leader pods in vidb-external
  ERROR : Script exited unexpectedly at line 19 with exit code 1

Environment

• VCF Operations 9.1.x
• VCF Services Runtime 9.1.x
• VCF Identity Broker 9.1.x

Cause

The failure is caused by a stale vidb-import-data secret that was left over from a previous 9.0 to 9.1 blue/green migration. During a standard 9.1.0 to 9.1.1 upgrade, the precheck script erroneously detects this leftover secret and extracts its outdated kubeconfig key. The script then attempts to query the old 9.0 source cluster rather than the active local cluster. Because the target PostgreSQL pods do not exist in the phantom source cluster, the script crashes.

Resolution

To resolve this issue, you must manually delete the stale migration secret so the precheck script queries the local cluster.

1. SSH into the VCF Services Runtime Control Plane using vmware-system-user
2. Switch to root using:

   sudo -i

3. Verify the existence of the stale import-data secret in the vidb-external namespace:

   kubectl -n vidb-external get secret | grep import-data
   
   (Expected output should list vidb-import-data)

4. Delete the stale secret:

   kubectl -n vidb-external delete secret vidb-import-data

5. Verify the secret has been successfully removed:

   kubectl -n vidb-external get secret | grep import-data
   
   (This should now return nothing)

6. Return to the VCF Operations UI and click Retry on the failed VCF Identity broker Upgrade Precheck Workflow and it should complete successfully.

Additional Information

• You can verify local VCF Identity Broker database health by executing the command below on VCF Services Runtime Control Plane. The output should show one Leader in a running state and Sync Standby replicas in a streaming state.

  kubectl exec -it vidb-postgres-instance-0 -n vidb-external -c postgres -- patronictl list