SSP Upgrade fails at 74%

Products

• VMware vDefend Firewall

Issue/Introduction

• SSP upgrade fails with this error message "Unable to get feature and access control information due to dependent services being unavailable. Consider checking the health status of the Kubernetes cluster, nodes, and pod you are using in your deployment." at 74%.
• SSP UI is unavailable.
• vSphere-csi-controller pod is crashlooping.
• Control plane node is stuck in provisioned state.
• spark app raw flow driver pod is not running.

Environment

SSP 5.0

Cause

This happens when vCenter certificate has changed or replaced, firewall port is not open and spark app rawflow is not running.

Resolution

To resolve this issue.

1) Ssh to SSP Installer as root and check the pod status.

k get pods -A | egrep -vi "runn|compl"

vmware-system-csi   vsphere-csi-controller-xxxxx                          6/7     CrashLoopBackOff        29254 (84s ago)    216d

Check the vsphere-csi-controller and vsphere-syncer  container logs, match the symptoms and apply the resolution from this KB 399170.

This will retrigger the recreation of the workload cluster nodes.

2) if Control plane nodes gets stuck in provisioned state, review this KB 423299. and open the firewall port as necessary.

3) Once SSP workload cluster is recreated and all the pods are running, SSP UI becomes accessible.

4) Continue with the upgrade, you see upgrade stuck at analytics on Intelligence page.

5) check the overflow and rawflow spark app status if it is running or not along with spark app raw flow and overflow pods.

k get pods -A | grep spark

k get sparkapp -n nsxi-platform

6) You see spark app rawflow in failing or invalidating status and spark app raw flow driver pod is not running.

7) To retrigger spark app, review this kb and update the core request as mentioned in this KB 387928.

8) Once rawflow driver is running, upgrade completes successfully.