Roll back and clean up after failed convergence of vSphere 8.x to VCF 9.1

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

Deployment fails during the "Configure Lifecycle Components" phase.

The VCF Installer UI or logs display one or more of the following:

PUBLIC_LCM_COMPONENTS_CONFIGURE_FAILED
VCF_LCM_500_INTERNAL_SERVER_ERROR
Could not open JPA EntityManager for transaction

Checking the SDDC LCM PostgreSQL cluster reveals that the primary leader node (vcf-sddc-lcm-db-1) is missing or destroyed, and the replica node (vcf-sddc-lcm-db-0) is in a start failed state.

PostgreSQL logs show a timeline divergence error similar to:

requested timeline 6 does not contain minimum recovery point 0/E00A278 on timeline 1

Environment

VCF 9.1

Cause

If underlying compute or resource instability causes the primary PostgreSQL database node to be destroyed during the initial deployment—before a valid backup or checkpoint is generated—the remaining replicas become stranded on a diverged timeline.

Without contiguous Write-Ahead Logs (WAL) from the primary node, the replicas cannot reach a consistent state to bootstrap a new leader, rendering the database irrecoverable.

 

There are many stages at which the deployment can fail and for many reasons. Below are a list of some of those.

• The process fails during the Deploy and configure VCF Management Platform phase (typically Step 4 of 14).

  • Steps to resolve are found in KB Failed to bootstrap VCF services runtime Reference Token: ####.

Resolution

Different scenarios may require different steps to revert and cleanup to prepare for fresh deployment of VCF 9.1. Please follow the steps below for your scenario.

Failed convergence from vSphere 8.x to VCF 9.1 at step 4 of 14.

1. From the VCF Installer download the existing deployment spec to avoid having to redo all of the configurations.

2. Power off and delete the VCF component VMs, including the VCF Installer that has already been used for the failed deployment..

3. Power off and delete the bootstrap VM.

4. Make the necessary changes to the JSON configuration file; for this scenario change the FQDN of all VCF components and Domain Name should be set to all lower case.

5. Deploy new VCF Installer.

6. Deploy via the JSON IIRC.

Note: Scripted vs. Manual cleanup:

1. Important Note for DB Corruption: If the deployment failure is due to the irrecoverable PostgreSQL timeline divergence mentioned above, the LCM database is completely offline. Because the database is unresponsive, standard scripted component cleanup methods will fail.

2. In this scenario, the only supported path is to execute the manual cleanup steps detailed in this KB: manually powering off and deleting the VCF Management Services runtime VMs, the Bootstrap VM, and the VCF Installer VM from disk.

3. Brownfield Caution: Emphasize that when performing this manual rollback in a brownfield convergence, administrators must take extreme care to only delete the VCF-created VMs and leave the existing vCenter, ESXi hosts, and NSX infrastructure untouched.