Error: "Backup workflow failed for component: ops-logs" during upgrade process of Log management 9.1

Products

• VCF Operations

Issue/Introduction

When attempting to upgrade Log management to version 9.1.1, the process fails during "Perform Log management upgrade" subtask status shows : Failed. The specific error messages displayed are:

Error: Backup workflow failed for component: ops-logs
Error: Step vmsp_upgrade failed. Reference Code: 27469C55. Check logs for details.

or

Error - Task timed out after 6h0m20s. 
Error - Step vmsp_prechecks failed. Reference Code: 6B1E1B7E. Check logs for details.

Environment

VCF Log management 9.1.x

VCF Operations 9.1.x

Cause

The failure often occurs due to issues during the automated backup process initiated by the upgrade workflow.

Resolution

Address Stalled Kubernetes Upgrade Workflows, If the upgrade task is stalled specifically at the "Perform Component Upgrade" stage for a long time, follow this flow to investigate and clear the backend Kubernetes workflows:

• • Check the workflows in running status. Refer to their age and correlate them with your original upgrade started window:

    • kubectl get workflows -A

  • Check the logs to gain insight into why the workflow has been in a running state for a long time (e.g., look for backupFailed or timedout errors):

    • kubectl logs -n vmsp-platform <workflow_name>

  • Post the above checks, proceed with deleting the workflow(s) which is in a long-running state and correlates to the stuck upgrade task:

    • kubectl delete workflow -n vmsp-platform <workflow_name>

 

Proceed with the below steps to resolve this issue, and manually trigger a backup for log management before retrying the upgrade.

1. Navigate in the VCF Operations interface to Build > Lifecycle > VCF Management.

2. Select Backup & Restore.

3. Click Backup Now.

4. Select the relevant VCF instance/component.

5. Allow the backup process to complete on the target component (log management in this case).

6. Once the backup is successful, retry the Log management upgrade.