Tanzu Kubernetes Grid Integrated Edition ingress fails to use secret in TLS

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:

• An ingress controller does not pick up the TLS secret defined certificate. Instead, it presents the nsx-lb certificate.
• You see messages similar to the following in the ncp/ncp.stdout.log file on the master node.This example used the secret named "test-secret". (Logs can be pulled via bosh logs cli command against the TKGI cluster deployment.)

1 2020-11-13T21:25:52.685Z 72186804-f26c-41a6-a55f-345f384aa02a NSX 7604 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="INFO"] nsx_ujo.ncp.inventory Inventory send out update [('CREATE', {'display_name': u'agreements', 'spec': 'rules:\n- host: ingress.example.local\n  http:\n    paths:\n    - backend:\n        serviceName: agreements\n        servicePort: 5050\ntls:\n- hosts:\n  - ingress.example.local\n  secretName: test-secret\n', 'container_cluster_id': '333321ca-b80a-ffda-b110-3234cda44f11', 'origin_properties': [], 'external_id': u'2bbaa11a-c662-5ffb-a429-7777aacddff1', 'container_project_id': u'a7f2c516-bab7-4215-a20c-e78772436c86', 'resource_type': 'ContainerIngressPolicy'})]
1 2020-11-13T21:25:52.860Z 72186804-f26c-41a6-a55f-345f384aa02a NSX 7604 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] vmware_nsxlib.v3.client The HTTP request returned error code 400, whereas 201/200 response codes were expected. Response body {u'error_code': 2038, u'error_message': u'Certificate already exists.', u'httpStatus': u'BAD_REQUEST', u'module_name': u'internal-framework'}
1 2020-11-13T21:25:52.860Z 72186804-f26c-41a6-a55f-345f384aa02a NSX 7604 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="INFO"] nsx_ujo.ncp.nsx.manager.nsxapi Attempted to import a certificate which has already been imported
1 2020-11-13T21:25:52.861Z 72186804-f26c-41a6-a55f-345f384aa02a NSX 7604 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] nsx_ujo.ncp.nsx.lb_l7_service Secret test-secret with the same PEM data has been imported, use a different secret instead

Environment

VMware PKS 1.x

Resolution

If the PEM used to create the secret has already been used in another secret or another cluster with the same NSX-T instance, NSX-T will not allow additional entries with the same PEM to be imported.

Currently sharing the same data in secrets is not supported for NSX ingress controllers. 

From the NSX-T Documentation for LoadBalancer CRDs to Handle Ingress Scaling

"Note that sharing the same secret data between different CRD load balancers is not supported. You must configure CRD load balancers with different certificates."

Workaround:
Create separate certificates for each ingress load balancer.