Resolving NSX-NCP Pod Stuck in CrashLoopBackOff due to NSX Search Service Error

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service
• VMware NSX

Issue/Introduction

• The NSX-NCP (NSX Container Plugin) Pod remains stuck in a CrashLoopBackOff state
• NSX UI shows an alarm for NCP plugin down
• Upon inspecting the logs of the nsx-ncp container of the NSX-NCP Pod, an error message indicating an issue with the NSX Search service is observed:

vmware_nsxlib.v3.exceptions.ManagerError: Unexpected error from backend manager (['#.#.#.#:443', '#.#.#.#:443', '#.#.#.#:443', '#.#.#.#:443']) for GET policy/api/v1/search/query?query=resource_type:IpAddressBlock&sort_by=id: Search service is currently unavailable, please restart using 'restart service search' and resync using 'start search resync all' CLI commands on the NSX Appliance

[ncp MainThread E errorCode="NCP00007"] nsx_ujo.common.utils NSX configuration error: Unexpected error from backend manager (['###-#-###-#####.###.###.##:443', '###-#-###-#####.###.###.##:443', '###-#-###-#####.###.###.##:443', '###-#-###-#####.###.###.##:443']) for GET policy/api/v1/search/query?query=resource_type:IpAddressPool AND tags.scope:ncp\/version AND tags.tag:#.#.# AND tags.scope:ncp\/cluster AND tags.tag:domain\-##\:#######\-#####\-####\-####\-############ AND tags.scope:ncp\/external AND tags.tag:true&sort_by=id: Search index is out of sync for 'resource_type:IpAddressPool'. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support. details: INDEXING_FAILURES_EXHAUSTED_RETRIES, params: [IpAddressPool,all]
[ncp MainThread C security="True" errorCode="NCP00250"] nsx_ujo.ncp.main NSX configuration validation failed

Environment

• VMware NSX
• VMware vSphere 7.0 with Tanzu
• vSphere Supervisor 8.0

Cause

A failure in the NSX Search service can cause the API call performed by the NSX-NCP Pod to the NSX Manager to fail, causing the NSX-NCP Pod to enter a CrashLoopBackOff state.

Resolution

Solution 1: Restart the NSX Search Service

Follow these steps to restart the NSX Search service and resolve the issue. If this solution doesn't resolve the problem, proceed to Solution 2.

1. Open an SSH session to every NSX Manager appliance VM as the admin user.
2. Restart the NSX Search service and resync the service by running the following commands on every NSX Manager appliance VM
   
   restart service search
start search resync all
   
   
3. Identify the NSX-NCP Pod. Run the following command from a system with kubectl installed and network access to the Kubernetes cluster.
   
   Note: If NCP is running on a vSphere supervisor, see the How to ssh into supervisor control plane VMs section of Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VMs for instructions.
   
   kubectl get pods -n vmware-system-nsx | grep nsx-ncp
   
   
4. Restart the NSX-NCP pod by deleting it, as it will be automatically recreated:
   
   kubectl delete pod <nsx-ncp pod name> -n vmware-system-nsx
   
   
5. After the NSX-NCP Pod is recreated, confirm it is now in a 'Running' state:
   
   kubectl get pods -n vmware-system-nsx | grep nsx-ncp

If the NSX-NCP Pod is now running without errors and doesn't enter CrashLoopBackOff after some minutes, the issue has been resolved. If the problem persists, proceed to Solution 2.

Solution 2: Restart wcp service/vCenter

1. Initially restart the wcp service on vCenter especially if restarting the vCenter is not a viable option first during the troubleshooting step. 
   
   service-control --restart wcp
   
   
2. Identify the NSX-NCP Pod. Run the following command from a system with kubectl installed and network access to the Kubernetes cluster.
   
   Note: If NCP is running on a vSphere supervisor, see the How to ssh into supervisor control plane VMs section of Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VMs for instructions.
   
   kubectl get pods -n vmware-system-nsx | grep nsx-ncp


3. Restart the NSX-NCP pod by deleting it, as it will be automatically recreated:
   
   kubectl delete pod <nsx-ncp pod name> -n vmware-system-nsx
   
   
4. After the NSX-NCP Pod is recreated, confirm it is now in a 'Running' state:
   
   kubectl get pods -n vmware-system-nsx | grep nsx-ncp


5. If not sufficient, reboot the vCenter Server instance where the supervisor cluster is running and rerun steps 2-4.

If the NSX-NCP Pod is now running without errors and doesn't enter CrashLoopBackOff after some minutes, the issue has been resolved.

Additional Information

• NSX-T UI not working - "Index out of sync, please resync via 'start search resync policy"
• Cannot fetch inventory details: Search service is currently unavailable