SSP UI inaccessible after DR restore fails due to a worker node restarting in the middle of the restore.

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

SSP UI inaccessible after DR restore fails due to a worker node restarting in the middle of the restore.

Environment

SSP 5.1

Cause

During an SSP restore, if a worker node restarts, it may result in the trust-manager deployment having zero replicas. When this happens, the trust-manager pod will not be running in the SSP cluster, which can impact essential services and lead to UI inaccessibility.

 

Steps to check trust-manager replica count:

1. SSH into the SSPI VM CLI as sysadmin user.

2. Run the following command:

k get deployment/trust-manager -o=jsonpath='{.spec.replicas}' -n nsxi-platform

3. If the output is 0, it indicates that the trust-manager deployment has no active replicas.

 

How to Check Relevant Logs:

SSH to the SSPI VM CLI as sysadmin user.

Run below command:

k get pods -n nsxi-platform | grep restore

View the logs of the restore job pod:

k logs <job-restore-#####-#####> -n nsxi-platform;

Where 'job-restore-#####-#####' is the name of the pod from step #2.

Resolution

1. Check if the trust-manager replica count is 0.

2. If so, run the following command to set the replica count to 1.

k scale deployment/trust-manager --replicas=1 -n nsxi-platform.

3. Wait for the system to stabilize, login to SSP UI and re-initiate the restore with the same backup.

If the system does not stabilize or login continues to fail, then redeploy SSP and re-initiate DR restore.