PKS cluster creation fails at prepare primary VM state with error "Insufficient resources to allocate in edge cluster"

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:

1. Enterprise PKS cluster creation fails with error "1 of 7 pre-start scripts failed."
2. You see messages similar to the following in /var/vcap/sys/log/pks-nsx-t-prepare-master-vm/pre-start.stderr.log file on the primary node

time="2019-06-05T07:54:36Z" level=error msg="Failed to createT1Router: &{ManagedResource:{RevisionedResource:{Resource:{Links:[] Schema: Self:<nil>} Revision:<nil>} CreateTime:0 CreateUser: LastModifiedTime:0 LastModifiedUser: SystemOwned:<nil> Description: DisplayName:lb-pks-2cd2885a-5226-421b-a253-4f9d54137fb6-cluster-router ID: ResourceType: Tags:[0xc0004da520]} AdvancedConfig:<nil> AllocationProfile:0xc000010108 EdgeClusterID:edfd4ddf-fed9-48a1-80b6-af8b3c466079 EdgeClusterMemberIndices:[] FailoverMode: FirewallSections:[] HighAvailabilityMode:ACTIVE_STANDBY PreferredEdgeClusterMemberIndex:<nil> RouterType:0xc0002f8670}" pks-networking=networkManager

Error: [POST /logical-routers][400] createLogicalRouterBadRequest  &{RelatedAPIError:{Details: ErrorCode:10087 ErrorData:<nil> ErrorMessage:[Routing] Insufficient resources to allocate in edge cluster EdgeCluster/edfd4ddf-fed9-48a1-80b6-af8b3c466079 and pool FORWARDING_POOL for context LogicalRouter/0aaffa8c-3747-4c8e-8fac-367ae2fac9b4. ModuleName:ROUTING} RelatedErrors:[]}

 

3. You see messages similar to the following in PKS cluster task

Last Action Description:  Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: 79eb65a9-c376-47ae-8a74-ba6b52a670ba, broker-request-id: a82e8c60-7d63-4e89-a8e0-40a767a3cc84, task-id: 1215349, operation: create, error-message: Action Failed get_task: Task 05399c05-8e22-4b91-6b85-0e17b500b0c0 result: 1 of 7 pre-start scripts failed. Failed Jobs:...

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware PKS 1.x

Resolution

Taking the Edges in and out of maintenance mode will resolve this issue and allow for the successful creation of Enterprise PKS clusters. Navigate to System, Fabric, Nodes, Edge Transport Nodes, select the Edge node, select Enter NSX Maintenance Mode from the Actions menu; select Exit NSX Maintenance Mode from the Actions menu.