Backup job fails with "backup-repo could not read repository data" due to corrupted backup repository (S3 bucket)

Products

• VCF Operations

Issue/Introduction

Fails to run a backup job for the ops-logs component with the following error:

The backup of component Operations Logs is failing due to API error. Check all Log Processor nodes are healthy.
Backup workflow failed for component: ops-logs

System reports a repository error stating that backup-repo could not read repository data. Investigate the log-processor pod logs, calls to the OpenSearch snapshot API (_snapshot/backup-repo) return an HTTP 500 Internal Server Error with an output similar to the following:

[https://log-store:9200], URI [_snapshot/backup-repo/backup-<timestamp>], status line [HTTP/1.1 500 Internal Server Error]
  {"error":{"root_cause":[{"type":"repository_exception","reason":"[backup-repo] could not read repository data from index blob"}]}}

Environment

• VCF Operations 9.1
• VCF 9.1 - Log Management

Cause

This issue occurs when the backup-repo (the S3 bucket used to store OpenSearch snapshots) becomes corrupted. The expected repository index blob goes missing or falls out of sync with the actual contents of the bucket (e.g., due to concurrent modifications or underlying storage issues). Consequently, OpenSearch disables the repository to prevent any further data corruption.

Resolution

To resolve this issue, remove the corrupted backup-repo configuration so it can be cleanly recreated, and then restart the backup processing.

Note: Apply this workaround only on the affected cluster's ops-logs namespace, and only when the exact error signature above matches.

1. Refer Establish an SSH Connection to a VCF Services Runtime Instance to ssh to control plane VM.
2. Disable drift detection on the bucket: Run the following command to prevent Flux from reconciling or reverting the manual changes you are about to make:

   kubectl annotate bucket ops-logs-bucket -n ops-logs helm.toolkit.fluxcd.io/driftDetection=disabled

3. Remove TTL from the bucket: Run the following command to prevent the bucket from expiring mid-recovery:

   kubectl -n ops-logs patch bucket.objectstorage.vmsp.vmware.com ops-logs-bucket --type=merge -p '{"spec":{"ttlDays":0}}'

4. Delete the corrupted backup repo: Run the following command to remove the OpenSearch snapshot repository registration:

   kubectl exec -it log-store-0 -n ops-logs -- sh -c \
      'OS_USER=$(cat /usr/share/opensearch/config/ops-logs-secrets/username)
       OS_PASSWORD=$(cat /usr/share/opensearch/config/ops-logs-secrets/password)
   curl --request DELETE --url https://localhost:9200/_snapshot/backup-repo -u "${OS_USER}:${OS_PASSWORD}" -k'

5. Restart the log-processor pod: Run the following command to restart the pod, which will recreate the bucket and repository registration:

   kubectl delete po log-processor-0 -n ops-logs

Additional Information

After a successful application of this workaround, the _snapshot/backup-repo endpoint will respond without a repository_exception, and backup jobs will complete successfully. The log-store and log-processor pods in the ops-logs namespace will remain in a Running/Ready state.