How to prevent NCP pod from creating duplicate virtual servers

Products

Pivotal CF Services

Issue/Introduction

Symptoms:

In the NSX-T manager you see duplicate virtual servers with the names ending in http and https_terminated.
In the NCP log, you see the entries similar to:

2018-08-17T22:49:36.140Z 8a339617-c86e-469c-8cfd-0c609104d53e NSX 7 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="ERROR" errorCode="NCP05010"] nsx_ujo.k8s.service_watcher Failed to trigger l4 loadbalancer service ADDED handler for service web2 in namespace default: Unexpected error from backend manager (['192.168.100.110']) for Loadbalancer service update : Failed to update loadbalancer service 59cee434-2db1-400c-bd3f-6870820e4573 with virtual servers [u'98dc54d8-573d-44f8-8396-bfb96118e0c4', u'59e6a99e-4d45-4f39-a6b5-95c8dfec9abb', u'3b56d349-6fbf-43bc-b9d7-6aa47119b7ed', u'6f23abcf-862e-4eec-a1f8-f82e49cd889a', u'5c2ea1f6-d6e1-46bb-aa82-b0aee577566a', u'ce605d7a-1001-4510-9be3-8dd947fc4164', u'd8fe77be-1d65-4b05-a39c-c17cff71fe94', u'29255b72-8846-4ec0-a9fa-0e7c6ec60d1c', u'50bff666-33f7-4659-802e-df1328b99413', u'47a24388-3656-4d5e-95a3-4af21fd37071', u'cceeec8d-5967-4b5c-a16e-2cd6d3ded148']: Unexpected error from backend manager (['192.168.100.110']) for PUT api/v1/loadbalancer/services/59cee434-2db1-400c-bd3f-6870820e4573 : Found errors in the request. Please refer to the relatedErrors for details. relatedErrors: Exceed maximum number of virtual servers on the Load balancer service. The maximum size of virtual servers for SMALL load balancer service form factor is 10, current size of virtual servers is 11.

Environment

VMware Pivotal Container Service 1.x

Cause

Until VMware PKS 1.1.4, it contains NCP 2.0 pod. This pod is responsible for performing managing resources in NSX-T, which includes creating routers, switches, allocating IPs, etc.

This issue occurs due to a known issue in NCP 2.0 where each time the pod is restarted, it will create duplicate virtual servers in NSX-T.

Resolution

This is a known issue affecting VMware PKS 1.1.4 and earlier.

This issue is resolved in VMware PKS 1.1.5, 1.2 and later, available at Pivotal Network.

Workaround:
To work around this issue:

Add tags to the Kubernetes cluster's load balancer:
1. Select your cluster's load balancer.
2. Click on Actions > Manage Tags.
3. Add the following tags:
  ncp/version: 1.0.0
  ncp/cluster: pks-<CLUSTER UUID>
  
  Note: The cluster UUID is the value of the pks/cluster tag.
Modify the Kubernetes nsx-ncp-config configmap :
1. Edit the NCP pod’s config map by running this command:
  kubectl edit configmap nsx-ncp-config -n pks-infrastructure
2. Comment out the line that has lb_service=lb-pks-<cluster_uuid>:
  Sample Output:
  # Name or UUID of the NSX lb service that can be attached by virtual servers
  #lb_service = <None>
  
  #lb_service = lb-pks-f6f760bb-6c22-48c8-9ce9-d3170f373945
3. Save the file.

Now if the NCP pod is restarted, it wont create the duplicate load balancers. Please note that if the worker node running the NCP gets re-created (through an upgrade/Bosh resurrection event, etc), the configmap needs to be updated again.

Additional Information

Gather NCP pod logs

1. Get the name of the NCP pod:

kubectl get pod -l component=nsx-ncp -n pks-infrastructure | grep nsx-ncp | cut -d' ' -f 1
nsx-ncp-6849b5f564-hfj6x

2. View the NCP pod logs

kubectl logs nsx-ncp-6849b5f564-hfj6x -n pks-infrastructure