LoadBalancer service stuck in pending state while creating a K8S cluster in VMware PKS 1.1.x

Products

• VMware

Issue/Introduction

Symptoms:

• You are unable to create a K8S cluster from VMware PKS 1.1.x

• Kubernetes ncp pod load balancer service stuck in pending.

Environment

VMware Pivotal Container Service 1.x

Cause

This issue occurs when you reach the virtual server limit on the LB. if the cluster is using SMALL LB, then limit is 10 Virtual Servers. Every port exposed by a service of type Loadbalancer gets a Virtual Server.

1. When a service of type Loadbalancer is created and if this leads to > 10 Total Virtual Servers, NCP starts deleting the Virtual Servers corresponding to this service, as we do not want to partially handle a service. Such a service will be in Pending state and this is expected.
2. But the deleted Virtual Servers from the above point are still erroneously tracked by NCP cache. This impacts services created later. Now if you create a service (which even though will not lead to exceeding Virtual server limit) will be in pending state. Problem will persist for all services created after point (2).
3. NCP restart, rebuilds this cache and hence resolves the problem. The pending services are handled properly and they come in Running state.

Resolution

This is a known issue in VMware PKS 1.1.x, when you reach the LB virtual server limit & cleanup some virtual server.

This issue is resolved in VMware PKS 1.2.x, available at Pivotal Network.

Workaround:

To work around this issue, restart the NCP service. This will rebuilds the cache and hence resolves the issue.