Troubleshooting "NetworkReservationNotEnoughCapacity" using the BOSH command line to determine IP usage

Products

• Operations Manager

Issue/Introduction

This article helps you to troubleshoot with BOSH CLI to determine IP usage for Operations (Ops) Manager V 2.0 and above.

When installing Add-On's or Tiles that require Compilation VM's on the BOSH subnet or VM's in one or more of the AZ subnets, you might encounter the following error:

NetworkReservationNotEnoughCapacity: Failed to reserve IP for ...  

There is an edge case when a BOSH add-on is installed and then removed, and that add-on creates VMs on the BOSH network. What happens is that the removal of the add-on does not destroy the VM's. As a result, the stale VMs holds onto the IP reservations and prevents it from being reused. If this happens enough or your subnet is small enough, you can run out of usable IP addresses.

The problem is further compounded because Ops Manager is conservative with the allocation of IP addresses from the unreserved range, and if it finds that an IP address is still in use it will not propagate changes made to the reserved-IP list.

Resolution

The BOSH command line can be used to determine if any assumed free IP's are reserved in the Ops Manager or availability zone (AZ) subnets that is preventing a successful deployment. You can take actions to either free unreleased IP's or create a new subnet if needed.

Follow these steps to verify what is in use:

1. Log into the BOSH VM.

2. Issue the following commands:

 - /var/vcap/jobs/director/bin/console

 - puts Bosh::Director::Models::IpAddress.map(&:info);nil 

:~$ /var/vcap/jobs/director/bin/console

=> Loading /var/vcap/jobs/director/config/director.yml
=> ruby-debug not found, debugger disabled
=> Welcome to BOSH Director console
=> You can use 'app' to access REST API
=> You can also use 'cloud', 'blobstore', 'nats' helpers to query these services

irb(main):001:0> puts Bosh::Director::Models::IpAddress.map(&:info);nil

p-redis-4cc84a909b61d9ec6837.broker-registrar/0 - env02 - 192.x.x.x (dynamic)
p-redis-4cc84a909b61d9ec6837.smoke-tests/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.consul_server/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.nats/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.etcd_tls_server/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.nfs_server/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.mysql_proxy/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.mysql/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.diego_database/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.uaa/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.cloud_controller/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.ha_proxy/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.router/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.mysql_monitor/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.clock_global/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.cloud_controller_worker/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.diego_brain/0 - env02 - 192.x.x.x (static)
cf-02aedca9addcd668a730.diego_cell/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.diego_cell/1 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.doppler/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.loggregator_trafficcontroller/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.smoke-tests/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.push-apps-manager/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.notifications/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.notifications-ui/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.push-pivotal-account/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.autoscaling/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.autoscaling-register-broker/0 - env02 - 192.x.x.x (dynamic)
cf-02aedca9addcd668a730.nfsbrokerpush/0 - env02 - 192.x.x.x (dynamic)
Pivotal_Single_Sign-On_Service-d500481727380914aa60.deploy-service-broker/0 - env02 - 192.x.x.x (dynamic)
p-windows-runtime-f21c6b3d70e14f3d9837.windows_diego_cell/0 - env02 - 192.x.x.x (dynamic)
p-windows-runtime-f21c6b3d70e14f3d9837.install-hwc-buildpack/0 - env02 - 192.x.x.x (dynamic)
p-redis-4cc84a909b61d9ec6837.cf-redis-broker/0 - env02 - 192.x.x.x (static)
p-redis-4cc84a909b61d9ec6837.dedicated-node/0 - env02 - 192.x.x.x (static)
p-redis-4cc84a909b61d9ec6837.dedicated-node/4 - env02 - 192.x.x.x (static)
p-redis-4cc84a909b61d9ec6837.dedicated-node/2 - env02 - 192.x.x.x (static)
p-redis-4cc84a909b61d9ec6837.dedicated-node/3 - env02 - 192.x.x.x (static)
p-redis-4cc84a909b61d9ec6837.dedicated-node/1 - env02 - 192.x.x.x (static)   

In the above example, you can take the output to confirm how many IP's are in use in the "env02" subnet.

If you sort on the 2nd column, you will better see the breakdown across all networks / AZ's.

In the case where you need to install a new Tile or Upgrade an existing tile, these deployments may need to create VM's in a subnet that has consumed all IP's or has them reserved in Ops Manager, so they will not be used.

If the Ops Manager network was defined with a limited number of IP's, such as a /27 that only has 30 IP's, and you reserve most of them or they are in use by the compilation VM's or other uses, then any Tile or Add-On that requires a VM on this subnet will run into this error. 

The recommendation is to use a service network for BOSH releases that are not installed by Ops Manager.