NCP constantly flapping and Container stuck in ContainerCreation

Products

• VMware NSX-T Data Center

Issue/Introduction

• On the NSX Manager, the number of IPSET is higher than expected, it could more than 10k and many of them are duplicates.
• Container creation is stuck "ContainerCreation" status.
• The following Contrainer creation failure can be seen in the container logs: "networkplugin cni failed to setup"
• In the NCP logs (ncp.stdout.log), similar message can be seen:

1 2021-05-28T16:41:22.021Z dde2e88e-####-####-####-5ba4552d1234 NSX 29162 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="CRITICAL"] nsx_ujo.ncp.main Failed to initialize container orchestrator adaptor: Unexpected error from backend manager (['mgr01.corp.local']) for GET api/v1/search?query=resource_type:IPSet AND tags.scope:ncp\/cluster AND tags.tag:pks\-12d12a1b\-####\-####\-####\-e7f79991234c&cursor=49000: The result set is too large. Please refine the search criteria

• In any container environment, that uses NSX Container Plugin (NCP), NCP is constantly restarting.
• In the NCP logs (ncp.stdout.log), the following error is reported:

The HTTP request returned error code 400, whereas 201/200 response codes were expected. Response body {'details': 'Field level validation errors: {display_name dst-k8s-111-n... has exceeded its maximum valid length 255}', 'httpStatus': 'BAD_REQUEST', 'error_code': 255, 'module_name': 'common-services', 'error_message': 'Field level validation errors: {display_name dst-k8s-111-n... has exceeded its maximum valid length 255}'}

Environment

VMware NSX-T Data Center

Resolution

This issue is resolved in VMware NSX-T Data Center 3.1.1

Workaround:

In order to avoid this issue:

• Limit the number of port-protocol per Network Policy and create more Network Policies.

If the issue is detected in the environment:

• Before NCP is constantly restarting
  • the network policy with many port-protcol (100+) can be deleted.
  • Then NCP will take care of cleaning up the IP-SETS creating in NSX-T Manager (Including the duplicate).
• NCP is constantly restarting
  • Contact Broadcom Support