Supervisor Update stuck at "Configuration error" with error "Component NSXNCPUpgrade failed"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When opening the current status of an ongoing Supervisor update, the status is at "Configuration error" for longer period of time (e.g., a 1-2 hours) without any progress
• When checking closer, the precise step failed is "Component NSXNCPUpgrade failed" with an error similar to:
  

  Component Configuration error: Component NSXNCPUpgrade failed: Failed to run command: ['kubectl', 'rollout', 'status', 'deployment', 'nsx-ncp', '-n', 'vmware-system-nsx', '--timeout=3m', '--watch=true'] ret=1 out=Waiting for deployment "nsx-ncp" rollout to finish: 1 old replicas are pending termination... err=error: deployment "nsx-ncp" exceeded its progress deadline Component upgrade failed.

• When verifying the current state of the NCP pods, they are in a healthy Running state:
  

  root@SV [ ~ ]# kubectl get pods -A | grep -i nsx
  vmware-system-nsx    nsx-ncp-649f6c84d5-6b68g        2/2     Running            0             3h35m
  vmware-system-nsx    nsx-ncp-649f6c84d5-qn9nl        2/2     Running            0             3h35m

• When verifying the state of other pods, it can also be seen "masterproxy-tkgs-plugin" not running: (due to not finding the expected image)
  

  root@SV[ ~ ]# kubectl -n svc-tkg-domain-c11 get pods -A|grep -v Running
  NAMESPACE            NAME                            READY   STATUS             RESTARTS       AGE
  svc-tkg-domain-c11   masterproxy-tkgs-plugin-577lx   0/1     ImagePullBackOff   0              109s
  svc-tkg-domain-c11   masterproxy-tkgs-plugin-ctwvq   0/1     ErrImagePull       0              109s
  svc-tkg-domain-c11   masterproxy-tkgs-plugin-wfp22   0/1     ImagePullBackOff   0              97s

• When verifying the current state of the ongoing upgrade, following can be observed:
  ("NSXNCPUpgrade" being in failed state, "WCPClusterCapabilities" being "upgraded" while all other steps are "pending") 
  

  root@SV [ ~ ]# /usr/lib/vmware-wcp/upgrade/upgrade-ctl.py get-status | jq '.progress | to_entries | .[] | "(.value.status) - (.key)"' | sort
  "failed - NSXNCPUpgrade"
  "pending - AKOUpgrade"
  "pending - AppPlatformOperatorUpgrade"
  "pending - CertManagerAdditionalUpgrade"
  "pending - CertManagerUpgrade"
  "pending - CRDMigration"
  "pending - CsiControllerUpgrade"
  "pending - EmbeddedTKGServiceCleanup"
  "pending - EnvPropsUpgrade"
  "pending - ExternalSnapshotterUpgrade"
  "pending - ImageControllerUpgrade"
  "pending - ImageRegistryUpgrade"
  "pending - KappControllerUpgrade"
  "pending - LicenseOperatorControllerUpgrade"
  "pending - LifecycleMgrUpgrade"
  "pending - LoadBalancerApiUpgrade"
  "pending - NamespaceOperatorControllerUpgrade"
  "pending - NetOperatorUpgrade"
  "pending - PinnipedUpgrade"
  "pending - PspOperatorUpgrade"
  "pending - RegistryAgentUpgrade"
  "pending - SchedextComponentUpgrade"
  "pending - SecretgenControllerUpgrade"
  "pending - SphereletComponentUpgrade"
  "pending - StoragePolicyQuotaUpgrade"
  "pending - TelegrafUpgrade"
  "pending - TMCUpgrade"
  "pending - UtkgClusterMigration"
  "pending - VmOperatorUpgrade"
  "pending - VMwareSystemLoggingUpgrade"
  "pending - WCPClusterCapabilitiesPostTMCUpgrade"
  "upgraded - WCPClusterCapabilities"

Environment

VMware vSphere Kubernetes Service

Cause

There can be various reasons like below where simple timeout can cause this symptom - 

• Rollout Timeout during Upgrade: The Supervisor upgrade fails to complete because a specific component, such as the NSX Network Control Plane (NCP), exceeds its progress deadline while waiting for the rollout status to finish.
• Component Deployment Failure: A configuration error occurs when the nsx-ncp deployment in the vmware-system-nsx namespace fails to reach the required number of available replicas within the defined timeout period.
• Client-Side Throttling: During the reconcile process of the Tanzu Kubernetes Release (TKR) service, excessive request delays occur due to client-side throttling (not priority and fairness), causing the upgrade to time out.
• Infrastructure Connectivity/Performance Issues: Intermittent delays in communicating with the API server during resource reconciliation lead to unsuccessful package installation and "ReconcileFailed" status for core supervisor services.

Resolution

For most simple timeout issues causing this symptom, retrying supervisor upgrade process through vCenter vSphere Client can resolve the issue. 

Once the upgrade gets retried, it'll continue where it left off and should lead to an upgraded state:

root@SV [ ~ ]# /usr/lib/vmware-wcp/upgrade/upgrade-ctl.py get-status | jq '.progress | to_entries | .[] | "(.value.status) - (.key)"' | sort
"skipped - CertManagerAdditionalUpgrade"
"skipped - CRDMigration"
"skipped - LoadBalancerApiUpgrade"
"upgraded - AKOUpgrade"
"upgraded - AppPlatformOperatorUpgrade"
"upgraded - CertManagerUpgrade"
"upgraded - CsiControllerUpgrade"
"upgraded - EmbeddedTKGServiceCleanup"
"upgraded - EnvPropsUpgrade"
"upgraded - ExternalSnapshotterUpgrade"
"upgraded - ImageControllerUpgrade"
"upgraded - ImageRegistryUpgrade"
"upgraded - KappControllerUpgrade"
"upgraded - LicenseOperatorControllerUpgrade"
"upgraded - LifecycleMgrUpgrade"
"upgraded - NamespaceOperatorControllerUpgrade"
"upgraded - NetOperatorUpgrade"
"upgraded - NSXNCPUpgrade"
"upgraded - PinnipedUpgrade"
"upgraded - PspOperatorUpgrade"
"upgraded - RegistryAgentUpgrade"
"upgraded - SchedextComponentUpgrade"
"upgraded - SecretgenControllerUpgrade"
"upgraded - SphereletComponentUpgrade"
"upgraded - StoragePolicyQuotaUpgrade"
"upgraded - TelegrafUpgrade"
"upgraded - TMCUpgrade"
"upgraded - UtkgClusterMigration"
"upgraded - VmOperatorUpgrade"
"upgraded - VMwareSystemLoggingUpgrade"
"upgraded - WCPClusterCapabilities"
"upgraded - WCPClusterCapabilitiesPostTMCUpgrade"