SSPI restore blocked/fails

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

SSPI restore blocked/fails with below errors:

The instance <instance_name> is unreachable. Please verify your network connection if the instance is active, or redeploy <instance_name> if it has been deleted.

Or
The instance <instance_name> is restored but unreachable. Please verify your network connection if the instance is active, or delete it from the SSP Installer if it no longer exists.

Or
The instance <instance_name> is reachable but there is a certificate mismatch. Please restore using the latest backup, or verify if the instance is running on another Installer.

Or
The instance <instance_name> is restored and reachable but there is a certificate mismatch. Please use the latest backup, or delete the instance from the SSP Installer if it is deployed elsewhere.

This article addresses a failure scenario where the SSPI Disaster Recovery (DR) restore workflow fails to start or complete with errors. This issue typically happens due to network environment mismatches, missing workloads, or conflicting security states between the backup metadata and the active environment.

Common instances as example involved in these errors include:

ssp-manager-node-1 (SSP Instance-1)
intelligence-appliance-1 (SSP Instance-2)

Error Messages

During an SSPI restore, one of the following error messages may be displayed in the UI or CLI logs:

The instance ssp-manager-node-1 is unreachable. Please verify your network connection if the instance is active, or redeploy ssp-manager-node-1 if it has been deleted.
The instance ssp-manager-node-1 is restored but unreachable. Please verify your network connection if the instance is active, or delete it from the SSP Installer if it no longer exists.
The instance intelligence-appliance-1 is reachable but there is a certificate mismatch. Please restore using the latest backup, or verify if the instance is running on another Installer.
The instance intelligence-appliance-1 is restored and reachable but there is a certificate mismatch. Please use the latest backup, or delete the instance from the SSP Installer if it is deployed elsewhere.

Environment

Security Services Platform(SSP) - 5.2.0

Cause

Connectivity Failures: The targeted instance is either deleted, powered down, or blocked by network firewalls, making it completely invisible to the installer.
Certificate Conflicts: The backup data is being restored into an environment where the instance is already registered to a different installer, or the localized certificates have changed, causing a secure handshake mismatch.

Resolution

WARNING: OPERATIONAL IMPACT & DOWNTIME HAZARD Deleting or redeploying an SSP instance (e.g., ssp-manager-node-1) is a high-risk operational task that results in management plane downtime.

Estimated recovery time: 1 to 3 hours depending on system scale.

Prerequisite: Ensure you have a full system backup and a scheduled maintenance window before executing destructive actions (such as instance deletion or redeployment).

Follow the recovery steps below based on the specific error scenario you are encountering:

Note: Please use the instance name as per your environment - as described in the introduction above, ssp-manager-node-1 and intelligence-appliance-1 mentioned below are taken as examples only.

Scenario 1: Instance is Unreachable

Applicable Error Messages:

The instance ssp-manager-node-1 is unreachable. Please verify your network connection if the instance is active, or redeploy ssp-manager-node-1 if it has been deleted.
The instance ssp-manager-node-1 is restored but unreachable. Please verify your network connection if the instance is active, or delete it from the SSP Installer if it no longer exists.

Recovery Steps:

If the ssp-manager-node-1 instance is active but network connectivity is broken:
- Troubleshoot and resolve the underlying network or routing issues between the installer and the instance.
- Here are the troubleshooting steps to confirm that SSP Instance is not accessible:
  - SSH to SSPI Appliance (Installer Node):
    ssh sysadmin@<SSPI_IP_OR_FQDN>
  - Check node accessibility:
    k get nodes --ssp ssp-manager-node-1
    Error: Get "https://192.###.##.#:6443/api/v1/nodes": dial tcp 190.###.##.#:6443: connect: connection refused OR Error: Get"https://192.###.##.#:6443/api?timeout=32s": dial tcp 192.###.##.#:6443: connect: no route to host
- Contact your network administrator for assistance.
- Once connectivity is restored, retry the SSP Installer restore operation.
- If connectivity issues persist, contact Broadcom administrator for assistance.
If you are sure that the ssp-manager-node-1 instance is completely lost or deleted:
- Here are the troubleshooting steps to confirm that SSP Instance is not accessible:
  - SSH to SSPI Appliance (Installer Node):
    ssh sysadmin@<SSPI_IP_OR_FQDN>
  - Run command:
    k get nodes --ssp ssp-manager-node-1
    Error: Get "https://192.###.##.#:6443/api/v1/nodes": dial tcp 190.###.##.#:6443: connect: connection refused OR Error: Get"https://192.###.##.#:6443/api?timeout=32s": dial tcp 192.###.##.#:6443: connect: no route to host
- Delete the instance ssp-manager-node-1 directly from the SSP Installer to clear its stale state in SSPI.
- Deploy a fresh ssp-manager-node-1 instance from this installer. Please refer SSP deployment guide to deploy a SSP instance.
- Perform a SSP restore operation specifically for ssp-manager-node-1. Please refer SSP restore guide to restore a SSP instance.

Note: In case a SSP instance connectivity has failed then it must be recovered or deleted from SSP Installer. The SSPI backup will be allowed only if all SSP instances are healthy.

Scenario 2: Certificate Mismatch Detected

Applicable Error Messages:

The instance intelligence-appliance-1 is reachable but there is a certificate mismatch. Please restore using the latest backup, or verify if the instance is running on another Installer.
The instance intelligence-appliance-1 is restored and reachable but there is a certificate mismatch. Please use the latest backup, or delete the instance from the SSP Installer if it is deployed elsewhere.

Recovery Steps:

If the intelligence-appliance-1 instance is still active on this environment:

Here are the troubleshooting steps to confirm that SSP Instance is not accessible:

SSH to SSPI Appliance (Installer Node):
```
ssh sysadmin@<SSPI_IP_OR_FQDN>
```

Check nodes:

k get nodes --ssp intelligence-appliance-1

Unable to connect to the server: x509: certificate signed by unknown authority

Fetch the live certificate fingerprint from the remote instance endpoint:

a. Execute:

k config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' --ssp intelligence-appliance-1 | base64 --decode | openssl x509 -noout -fingerprint -sha256

SHA256 Fingerprint=TH:IS:XA:SA:MP:LE:FI:NG:ER:PR:IN:TX:1A:2B:3C:4D:5E:6F:7G:8H:9I:0J:7D:6C:5B:4A

b. Query the remote kube-root-ca.crt by temporarily bypassing TLS verification:

Execute:

k get configmap kube-root-ca.crt --ssp intelligence-appliance-1 --insecure-skip-tls-verify -o jsonpath='{.data.ca\.crt}' | openssl x509 -noout -fingerprint -sha256

SHA256 Fingerprint=TH:IS:XA:SA:MP:LE:FI:NG:ER:PR:IN:TX:1A:2B:3C:4D:5E:6F:7G:8H:9I:0J:7D:6C:5B:4A 

Fingerprint do not match.

Attempt a restore using a more recent SSP Installer backup that contains the matching certificate pair.
If a backup with matching certificates is unavailable, delete the current intelligence-appliance-1 instance.
Redeploy the intelligence-appliance-1 instance, and then perform a standalone SSP restore operation on it.

If you are sure that the intelligence-appliance-1 instance was deleted or moved to a different installer node:
- Delete the intelligence-appliance-1 instance from your current installer node to remove the conflicting configuration.

Note: In case a SSP instance connectivity has failed then it must be recovered or deleted from SSP Installer. The SSPI backup will be allowed only if all SSP instances are healthy.