Error: "PostgreSQL version upgrade failed for instance 'vcd-migrator-postgres' in namespace 'vcd-migrator'" when attempting to upgrade migration service engine in VCF 9.1.1

Products

• VCF Operations
• VMware Cloud Foundation

Issue/Introduction

Upgrading VCF management components to version 9.1.1 fails specifically for the migration service engine component.

The issue typically presents under the following conditions:

• The initial upgrade attempt for the migration service engine was performed prior to successfully upgrading VCF Automation to 9.1.1.
• Reattempting the upgrade after VCF Automation is successfully on 9.1.1 continues to fail.
• The migration service engine displays the following error:
  
  PostgreSQL version upgrade failed for instance 'vcd-migrator-postgres' in namespace 'vcd-migrator'. Please retry the operation or contact support if the issue persists. [VCFMS-PGHOLD-003]
  
  
• Running "kubectl -n vcd-migrator get postgresinstance vcd-migrator-postgres -o yaml" reveals that while the PostgreSQL instance successfully deployed and upgraded from version 14 to 17, it exhibits a post-upgrade validation failed:
  
  database ping failed after upgrade message.



• Attempting to use the UI's "retry" option results in a 404 Not Found error:
  
  An unexpected error occurred in step vmsp_upgrade. Reference Code: 871F3EE4. Please contact support with this reference code. Detail: 404 Not Found: "{"messages":[{"level":"ERROR","id":"com.vmware.vcfms.api.NotFound","default":"Not Found"}]}<EOL>"
  
  

Environment

VCF 9.x

Cause

The upgrade failure is the result of two issues:

1. The PostgreSQL database successfully upgraded from version 14 to 17 and actively accepts connections on port 5432, but a post-upgrade validation ping failed. This leaves an upgrade-failed annotation on the postgresinstance that prevents subsequent operations from proceeding.
2. The "Retry" button fails with a 404 Not Found error because the original upgrade task job is older than 20 days and has been automatically cleared from the backend tasks. Running "kubectl -n vmsp-platform get jobs" will confirm the job is no longer running.

Resolution

To resolve the issue, back up the configuration, clear the failure annotation, and initiate a full upgrade rather than a task retry.

Step 1: Backup Current Configuration

Create a backup of the current state before making modifications.

1. SSH to one of the control plane VMSP runtime services VMs. Reference TechDoc Establish an SSH Connection to a VCF Services Runtime Instance for details.

2. Run the following commands:

   NS=vcd-migrator
   PG=vcd-migrator-postgres
   POD=vcd-migrator-postgres-0
   CASE_DIR=$(mktemp -d "$HOME/ubmfmp-27105.XXXXXX")
   kubectl -n "$NS" get postgresinstance "$PG" -o yaml > "$CASE_DIR/postgresinstance-before.yaml"
   kubectl -n "$NS" get postgresql "$PG" -o yaml > "$CASE_DIR/postgresql-before.yaml"
   kubectl -n "$NS" get endpoints "$PG" -o yaml > "$CASE_DIR/endpoints-before.yaml"
   kubectl -n "$NS" exec "$POD" -c postgres -- patronictl list > "$CASE_DIR/patroni-before.txt"
   kubectl -n "$NS" logs "$POD" -c postgres --since=1h > "$CASE_DIR/postgres-before.log"
   kubectl -n vmsp-platform logs deploy/vmsp-operator --since=1h > "$CASE_DIR/operator-before.log"
   

Step 2: Clear the Failure Annotation

Remove the upgrade-failed annotation from the PostgreSQL instance so the service can reconcile.

kubectl -n "$NS" annotate postgresinstance "$PG" database.vmsp.vmware.com/upgrade-failed-

Note: The trailing hyphen (-) in the command above is required as it removes the annotation.

Step 3: Verify PostgreSQL Health

Confirm the annotation is cleared and that the database is healthy and running on the new version.

# Verify instance conditions (UpdatingInstance should be False, Ready should be True)
kubectl -n "$NS" get postgresinstance "$PG" -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'

# Verify PostgreSQL server version (Should output 17.x)
kubectl -n "$NS" exec "$POD" -c postgres -- psql -U postgres -tAc 'SHOW server_version; SELECT pg_is_in_recovery();'

Step 4: Perform a Full Upgrade

If the original task has expired (over 20+ days ago), use the upgrade option. Note: If the original upgrade task has not yet expired, use the retry option.

1. Select the Upgrade option for the migration service engine in the VCF Management UI to begin a new process.

2. Monitor the staging progress via the CLI:
   
   kubectl -n vmsp-platform get task

3. The UI will display the status: Stage VCD_MIGRATOR plugin in VCF services runtime.

4. Once the component successfully upgrades to 9.1.1, an automatic inventory sync triggers (Inventory sync post VCD_MIGRATOR upgrade).

Note on UI Status: During this process, the Fleet Mgmt Instance status may display:

This component instance is waiting for an active task or dependent upgrade to finish. It will continue automatically once complete.

If this message appears, refresh the UI and the message will go away.