Log Management does not come back online after restoring it to a new management services cluster.
search cancel

Log Management does not come back online after restoring it to a new management services cluster.

book

Article ID: 436904

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

Pre-Requisite

  • Ensure you have access to the VCF Management Services control plane node and are able to copy files to it Refer here for details.
  • Check if you had NFS storage configured in the previous cluster using the below command. If yes then please ensure that the NFS storage path is accessible from the new cluster before moving forward. Please reach out to support if you cannot make the attached NFS accessible.
kubectl get packagedeployment operations-logs -n ops-logs -o yaml | grep -E '^[[:space:]]*(nfsStorage|nfsStorages):[[:space:]]*.+$'

 

This document describes how to recover the Log Management component to a healthy, ingest-ready state after disaster recovery when a backup taken on a previous cluster is restored onto a new or rebuilt VMware Cloud Foundation management cluster.

Use the supported recovery bundle and script to reconcile Log Management post restore.

Symptoms

No Log ingestion showing on Analyze page.

Environment

VMware Cloud Foundation (VCF 9.1)

Resolution

Step 1 — Obtain the recovery bundle

Download the recovery bundle and verification artifacts from your Broadcom-published location for this KB 142814:

  • Bundle: <URL or portal path to opslogs-recovery.tar.gz>
  • Detached signature: <URL or path to .sig file> (example filename from packaging: 4f29a5b0309d0d31208aab206e5b6c5a.sig)
  • Public key: <URL or path to public_key.asc> used to verify the signature

Note: Exact filenames may change per release; use the names supplied with your download package.

Step 2 — Import the signing public key (one time per workstation)  

gpg --import public_key.asc

Confirm the key belongs to the publisher listed in the KB or release notes.

Step 3 — Verify the tarball before use

Verification proves the tarball was not altered after signing. Replace filenames if your package uses different names.

gpg --verify 4f29a5b0309d0d31208aab206e5b6c5a.sig opslogs-recovery.tar.gz

Expected: Good signature from … for the correct key.

 

Troubleshooting:

Issue

Action

Can't check signature: No public key

Import the correct public key that matches the signing key ID shown in gpg --verify output; do not substitute unrelated keys (e.g. VMware Linux packaging keys unless documented for this bundle).

Signature bad

Re-download the bundle and .sig; do not extract or run unverified archives.

Step 4 — Copy and Extract the bundle

Copy the downloaded tar file to VMSP control plane node.

scp opslogs-recovery.tar.gz vmware-system-user@<CONTROL_PLANE_IP:~
tar -xzf opslogs-recovery.tar.gz
cd <extracted-directory>

Step 5 — Follow README and run the CLI

Open README.md in the extracted directory and follow the instructions exactly.

Summary (see README for authoritative flags and behavior):

Mode

When to use

--post-restore

Default after infrastructure restore when the log ingestion is failing. Reconciles credentials and rebuilds storage layout without calling the restore API again.






--restore

Only if a restore task shows failed on the component page.. Requires breakglass credentials; performs post-restore steps and re-initiates restore. See screenshot below to check restore task status on ui.

Examples (from product README — adjust paths to match extraction):

./opslogs-cli.sh --post-restore
./opslogs-cli.sh --restore -u '<breakglass-user>' -p '<breakglass-password>'

The tool prompts for confirmation before destructive steps; read warnings carefully. For failures, re-run with verbose logging and contact Broadcom Support with full output.

Additional Information

After a successful run:

  • Pods: log-store and log-processor workloads in namespace ops-logs reach Running/Ready where applicable.
  • Ingestion: New logs appear in Log Management search within expected latency.
  • UI / integration: VMware Cloud Foundation Operations shows Log Management installed and integrated as expected for your release.

Use your standard health checks (logs health dashboard sample queries) per your runbook.