Diego Cell scaling failures due to "acquire subnet lease: http status 409: No lease available" error

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This article discusses Apply Change failures during a scale up in the number of Diego cells.

Below are the errors from the Diego cell logs, the following errors are logged in the silk-daemon.stdout.log and silk-daemon.stderr.log:

silk-daemon.stdout.log:

{"timestamp":"1520997730.240546465","source":"cfnetworking.silk-daemon","message":"cfnetworking.silk-daemon.http-client",
"log_level":2,"data":{"body":"{\"error\": \"No lease available\"}","code":409,"error":"No lease available"}}

silk-daemon.stderr.log:

2018/03/14 03:22:10 cfnetworking.silk-daemon error: acquire subnet lease: http status 409: No lease available 

By default, the 10.255.0.0/16 subnet CIDR range is used for the Overlay Subnet for container networking.

The subnet IP addressing scheme for container networks is also changed when container networking is enabled. In the new addressing scheme, the 4th octet of the subnet is reserved for containers per Diego cell. This equals 254 containers per Diego cell. The first 3 octets are for the number of Diego cells.

Thus, if 10.255.0.0/16 is configured, and since the last octet is used for containers per cell, the bits from the 3rd octet that is available as part of the /16 CIDR range will only accommodate 255 Diego cells. When more than 255 Diego cells are configured in TAS, the “Apply Changes” will fail with the above errors. This is because the number of Diego cells are limited by the subnet selection for the "Overlay Subnet". See the table below on how the subnet CIDR range for the Overlay Subnet will affect cell capacity in TAS:
 

Overlay Subnet CIDR	Number of cells	Containers per cell
/22	3	254
/20	63	254
/16	255	254
/12	4095	254

Environment

Product Version: 2.7

Resolution

Under Elastic Runtime or the TAS / PAS tile, go to settings and networking, check the Overlay Subnet box. If this subnet is configured for 10.255.0.0/16, override the default subnet of 10.255.0.0/16 to either 10.240.0.0/12 or a subnet range that adheres to RFC 1918 and click Apply Changes.

Additional information: The count for number of diego_cells able to run is a combination of Linux Diego Cells and Isolated Segment Linux Diego Cells.