Troubleshooting SSP 5.0.0 to 5.1.1.1 Upgrade Failure: Upgrade Coordinator and Trust Manager Synchronization

Products

• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

During the platform stage of the SSP 5.0.0 to 5.1.1.1 upgrade, critical components undergo upgrades. This process causes the SSP UI to become momentarily unavailable.

Upon logging back into the SSP UI and navigating to the Upgrade page, a "something went wrong" error is displayed. Further investigation is required to determine the state of the Upgrade Coordinator:

1. Check Pod Status: Run the following command to identify the status of the Upgrade Coordinator pod: k get pods -n nsxi-platform | grep upgrade-coordinator The pod may show as 0/1 Running or ContainerCreating.

2. Inspect Pod Events: Run the following command to view events for the specific pod: k describe pod <name_of_the_upgrade_coordinator_pod> -n nsxi-platform Look for warnings similar to the following: Warning FailedToRetrieveImagePullSecret 17m (x6 over 19m) kubelet Unable to retrieve some image pull secrets (regcred); attempting to pull the image may not succeed.

3. Review Pod Logs: Run the following command to check the pod logs for the keystore loading error: k logs <name_of_the_upgrade_coordinator_pod> -n nsxi-platform Logs may indicate a failure to load the keystore: java.io.IOException: DerValue.getBigIntegerInternal, not expected 48 while loading from /certs/truststore.p12 with type pkcs12.

Environment

SSP 5.0

SSP 5.1.1.1

Cause

The trust-manager service failed to properly update or sync certificates during the platform upgrade, resulting in a corruption or mismatch of the truststore.p12 file. Because the Upgrade Coordinator relies on this truststore for API communication, the service fails to initialize, leading to the observed deployment error in the SSP UI.

Resolution

To resolve this synchronization failure and resume the upgrade, perform the following steps:

1. Restart Trust Manager: Perform a rolling restart of the trust-manager deployment to force a re-synchronization of certificates: k rollout restart deployment trust-manager -n nsxi-platform

2. Verify Status: Wait for the trust-manager pod to return to a Running state before proceeding.

3. Undeploy Upgrade Coordinator: Send an API request to undeploy the currently stuck Upgrade Coordinator feature:

   • Endpoint: POST https://{{ssp}}/napp/api/v1/platform/cluster-api/features/upgrade-coordinator
   • Body: {"action": "UNDEPLOY"}

4. Redeploy Upgrade Coordinator: Navigate to the Upgrade page in the SSP UI and redeploy the Upgrade Coordinator feature.

5. Verification:

   • Confirm the upgrade-coordinator pod status transitions to 1/1 Running.
   • The upgrade process for downstream components—including Metrics, Intelligence, NDR, and MPS—should resume from appropriate state.