Log management upgrade is stuck on Install VCF services runtime when upgrading from 9.0.2 to 9.1

Products

• VCF Operations

Issue/Introduction

When you upgrade log management from version 9.0.2 to 9.1, you observe that the upgrade task gets stuck and fails to complete. Specifically, the task hangs on the Install VCF services runtime step.

When you review the components page, the VCF services runtime appears to be successfully deployed, and log management is running on it. Additionally, the older 9.0.2 log VMs are still running in the vCenter Server and their UI is accessible alongside the newly deployed logs component.

Environment

VCF Operations 9.1.0

Cause

The upgrade process actually finishes successfully in the background. However, the Fleet Upgrade Service loses connection to the database. As a result, it loses track of the SDDC Upgrade Service's status, causing the UI to permanently display a stuck state because the status is never updated.

Resolution

To resolve this issue and clear the stuck upgrade plan, restart the Fleet and SDDC Java services on the VSP Control plane node:

1. Open an SSH session to the VSP Control plane node and log in as root.
2. Run the following command to export the KUBECONFIG:

   export KUBECONFIG=/etc/kubernetes/admin.conf

3. Run the following command to retrieve the SDDC pods:

   kubectl get pods -n vcf-sddc-lcm

4. Using the exact pod names returned from the previous output, delete the SDDC build and upgrade pods:

   kubectl delete pod vcf-sddc-build-<copy-full-name-from-above-output> -n vcf-sddc-lcm
   kubectl delete pod vcf-sddc-upgrade-<copy-full-name-from-above-output> -n vcf-sddc-lcm

5. Run the following command to retrieve the Fleet pods:

   kubectl get pods -n vcf-fleet-lcm

6. Using the exact pod names returned from the previous output, delete the Fleet build and upgrade pods:

   kubectl delete pod vcf-fleet-build-<copy-full-name-from-above-output> -n vcf-fleet-lcm
   kubectl delete pod vcf-fleet-upgrade-<copy-full-name-from-above-output> -n vcf-fleet-lcm

7. Wait for a couple of minutes for the services to fully restart. The stuck upgrade plan then disappears from the UI.