Error : The backup of component Operations Logs is failing due to API error. Check all Log Processor nodes are healthy during the Log management upgrade to 9.1.0.x

Products

• VCF Operations

Issue/Introduction

• Attempting to upgrade log management to version 9.1.0.x, the log management upgrade workflow fails. The "perform log management upgrade" task fails with the following errors in the user interface:
  
  

  Error The backup of component Operations Logs is failing due to API error. Check all Log Processor nodes are healthy
  info SFTP is configured, backup encryption enabled=true.
  info Database backup has been completed successfully for component ops-logs.
  info Backup has been completed successfully for component ops-logs.
  Error  Backup workflow failed for component: ops-logs
  Error  Step vmsp_upgrade failed. Reference Code: FA4151C0. Check logs for details.

  

  
• The console.log located at /services-logs/ops-logs/<VCFMS_Node_Name>/log-processor-0/vcf-ops-logs/console.log displays the following exception:
  
  

  org.opensearch.repositories.RepositoryException: [backup-repo] Could not read repository data
   because the contents of the repository do not match its expected state... The repository has
   been disabled to prevent corrupting its contents.
     at BlobStoreRepository.corruptedStateException(BlobStoreRepository.java:3175)
     at BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:3005) 

  
  
  
• The console.log located at /services-logs/vcf-sddc-lcm/<VCFMS_Node_Name>/vcf-sddc-upgrade-service-sddcupgrade-####/sddcupgrade/console.log displays the following errors:
  
  

  ERROR sddc-upgrade-service ## [lcm@#### scPath="ops" thread="system-task-worker-##" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="logTaskMessage:##"] [] - Step vmsp_upgrade failed. Reference Code: FA4151C0. Check logs for details.
  ERROR sddc-upgrade-service ## [lcm@#### scPath="ops" thread="system-task-worker--##"" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="appendRefCode:###"] [] - (FA4151C0) (com.broadcom.vcf.lcm.task.orchestrator.AsyncStepFailed) Step vmsp_upgrade failed. Reference Code: FA4151C0. Check logs for details.
  INFO sddc-upgrade-service ## [lcm@#### scPath="ops" thread="system-task-worker--##"" logger="c.n.c.c.execution.WorkflowExecutor" method="decide:1132"] [] - Execution terminated of workflow: VSP_COMPONENT_UPGRADE.1/<Task_ID>.RUNNING
  stdout F com.netflix.conductor.core.exception.TerminateWorkflowException: Task <Id> failed with status: FAILED and reason: 'null'

   

Environment

VCF Operations 9.1.x

Cause

This issue is caused by backup repository corruption within OpenSearch. The corrupted state forces the repository to be disabled, which prevents the backup task from completing successfully during the upgrade workflow.

Resolution

This issue will be resolved in the upcoming VMware Cloud Foundation 9.1.1 release.

To work around this issue in VMware Cloud Foundation 9.1, the following commands must be executed to remove the corrupted repository state and unblock the upgrade.

Run the steps on one of the control plane nodes. (go to VCF Operations -> Build -> Lifecycle -> Component -> VMware Runtime Services -> at the bottom there will be all VMSP nodes, connect to any one of the control plane nodes using vmware-system-user)

Export the config: 

export KUBECONFIG=/etc/kubernetes/admin.conf 

1.Disable drift detection for the bucket:

kubectl annotate bucket ops-logs-bucket -n ops-logs helm.toolkit.fluxcd.io/driftDetection=disabled

2.Remove the TTL from the bucket:

kubectl -n ops-logs patch bucket.objectstorage.vmsp.vmware.com ops-logs-bucket --type=merge -p '{"spec":{"ttlDays":0}}'

3.Delete the corrupted Backup Repository:

kubectl exec -it log-store-0 -n ops-logs -- sh -c '
  OS_USER=$(cat /usr/share/opensearch/config/ops-logs-secrets/username)
  OS_PASSWORD=$(cat /usr/share/opensearch/config/ops-logs-secrets/password)
  curl -w "\n%{http_code}\n" --request DELETE \
    --url https://localhost:9200/_snapshot/backup-repo \
    -u "${OS_USER}:${OS_PASSWORD}" -k'

4.Restart a log-processor pod to apply changes:

kubectl delete pod log-processor-0 -n ops-logs

Once the pod has restarted and stabilized, the upgrade workflow can be safely retried.