upgrade-coordinator pod deployment stuck at 9 % while upgrade SSP from 5.1.0 to 5.1.1

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

upgrade-coordinator pod  deployment stuck at 9 %  while upgrade SSP from 5.1.0 to 5.1.1 and observed below error in SSP UI -> system -> upgrade -> prepare for upgrade page as below

Environment

SSP >  = 5.1

Cause

This issue could be  due to trust-manager is not provisioning certificates

please follow the below steps to verify  Upgrade Coordinator pod is stuck in the ContainerCreating state, 

1. Log in to SSPI using the sysadmin credentials.

2. Check the Upgrade Coordinator pod status by running:
   k -n nsxi-platform get pods | grep upgrade-coordinator

3. Identify the Upgrade Coordinator pod name from the above output (the one stuck in ContainerCreating state).

4. Check the events for that specific pod to understand the issue:
   k -n nsxi-platform describe pod <upgrade-coordinator-pod-name>
   (Use the exact pod name obtained from Step 2.) 

please refer below  sample output: 

k -n nsxi-platform get pods | grep upgrade-coordinator
upgrade-coordinator-7cb8fbff68-q9d8w                              0/1     ContainerCreating   0                5h15m
upgrade-coordinator-install-wcqcj-xc8dj                           0/1     Error               0                5h31m
upgrade-coordinator-install-wcqcj-zvd52                           0/1     Error               0                6h1m

# k -n nsxi-platform events --for pod/upgrade-coordinator-7cb8fbff68-q9d8w
LAST SEEN                 TYPE      REASON        OBJECT                                     MESSAGE
3m29s (x163 over 5h18m)   Warning   FailedMount   Pod/upgrade-coordinator-7cb8fbff68-q9d8w   MountVolume.SetUp failed for volume "certs" : references non-existent secret key: truststore.p12

Resolution

If the Upgrade Coordinator pod is stuck in the ContainerCreating state, please follow the below steps:

1. Restart the Trust Manager pod to resync the certificates:

   • First, identify the Trust Manager pod:
     k get pods -A | grep trust-manager

   • Note the pod name from the output.

   • Restart the Trust Manager pod:
     k -n nsxi-platform delete pod <trust-manager-pod-name>

2. Wait for the Trust Manager pod to come back up and reach the Running state.

3. Once the Trust Manager pod is up, verify the Upgrade Coordinator pod status again:
   k -n nsxi-platform get pods | grep upgrade-coordinator

The Upgrade Coordinator pod should now come up successfully and if pod is up check the pod status in SSP -> upgrade-> prepare UI  and it should mark as Upgrade Coordinator deployment is completed .

If the issue still persists after performing the above steps, please contact Broadcom Support for further investigation and resolution.