NCP agent fails with error "IP block is exhausted to allocate subnet"

Products

• VMware NSX-T Advanced for VMware Tanzu Application Service Term License (50 pack Application Instance)
• VMware Tanzu Application Service
• VMware Tanzu Application Service

Issue/Introduction

NSX-T for TAS requires configuring IP blocks for use by NCP tile. This IP block is first configured on NSX Manager as detailed in Configure Gateways tile documentation. Then the IP Block is configured in the IP Blocks of Container Networks field NCP pane as detailed in Install and configure the NSX-T container plug-in documentation.

Container subnets (typically smaller of /24 or /25 size) are carved out of the larger IP block (typically of /16 or /17) size. When the IP block is exhausted on NSX manager, then API requests from NCP agent will begin getting "IP block is exhausted to allocate subnet" errors.

ncp.stdout.log on Diego BBS VM:

ncp/ncp.stdout.log:2024-07-11T13:20:43.162Z <GUID> NSX 8336 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="DEBUG"] nsx_ujo.ncp.inventory Update cluster network err: ('ContainerCluster', '<GUID>'), <GUID>, UNHEALTHY, [{'error_message': 'IP block is exhausted to allocate subnet', 'error_code': 'NCP00017', 'spec': '{"origin": "<GUID>"}'}]

Environment

TAS for VMs with NSX-T Networking. This KB potentially apply to all versions.

Cause

The error "IP block is exhausted to allocate subnets" means that the IP block on NSX Manager is full. NSX Manager is unable to allocate any further subnets for the container networking. You can view the allocated subnets under the IP block in NSX Manager.

The error may also occur if the IP block configured within NCP is removed or suddenly changes. 

Resolution

The most likely cause is that IP block on NSX Manager is exhausted. Either add a new IP block or expand the size of the existing one.

You should note the CIDR size of the IP block and container subnets in NSX manager, as well as the total number of subnets created under IP block.

For example a /16 IP block supports 65,536 IP addresses and /25 container subnet uses 128 IP's each. So after 512 subnets (65,536 / 128) are allocated then the IP block will be exhausted.

Also, note when an application instance is deleted that it takes 2 minutes for the IP address to be released. If there's a large influx of delete and recreates of application instances then this may lead to IP block exhaustion more quickly. So this needs to be considered when capacity planning for container networks.

Unused organizations can also potentially be removed to reclaim subnets under an IP block.

It's also recommended to check recent changelog's under OpsManager to ensure that IP block configuration NCP tile did not change recently.