NCP flapping with error "Failed to initialize container orchestrator adaptor"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

NCP is continuously flapping and the following error is observed in ncp.stdout.log

NSX 7728 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="CRITICAL"] nsx_ujo.ncp.main Failed to initialize container orchestrator adaptor: Unexpected error from backend manager (['']) for GET api/v1/search?query=resource_type:IPSet AND tags.scope:ncp\/cluster AND tags.tag:pks\-62d15a9b\-e461\-####\-####\-e7f79191800c&cursor=49000: The result set is too large. Please refine the search criteria.

It is failing to retrieve the list of IPSets from NSX-T Manager.

As NCP is not stable, other symptoms can include containers stuck in ContainerCreating phase with error:

networkPlugin cni failed to set up pod "<POD NAME>" network: netplugin failed with no error message

Environment

Product Version: 1.7

Resolution

The number of IPSets on NSX-T has exceeded the limit of 10000.
Check the number of IPsets on NSX-T
curl -k https://'USERNAME:PASSWORD'@NSX-T-MANAGER/api/v1/ip-sets | jq {result_count}

If it is great than 10000, then please engage NSX-T Support team to identify what triggered the creation of these IPsets and if any cleanup can be performed.
Please provide NSX-T log bundle, NSX-T version and NCP version to NSX-T team.