TKGi NSX ingress serving default NSX LB instead of provided certificate

Products

VMware Tanzu Kubernetes Grid Integrated Edition VMware Tanzu Kubernetes Grid Integrated Edition (Core) VMware Tanzu Kubernetes Grid Integrated Edition 1.x VMware Tanzu Kubernetes Grid Integrated (TKGi)

Issue/Introduction

When you try to verify the ingress Serving certificate, you get the NSX certificate instead and it is defined in tls secret:

openssl s_client -showcerts -CAfile FWCARoot.cer -connect <FQDN>:443 
CONNECTED(00000003) depth=0 CN = nsx-lb verify error:num=18:self signed certificate verify return:1 depth=0 CN = nsx-lb verify return:1

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nsx"
    ingress.kubernetes.io/rewrite-target: /
  name: test1
spec:
  tls:
    - secretName: test
  rules:
  - host: <FQDN>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-ping
            port:
              number: 80

When verifying the certificate with openssl it is valid and return the valid and expected results:

openssl x509 -text -noout -in server.cert

From NCP logs :

2024-07-11T07:41:57.892Z 944293c3-6624-47ef-8d97-9612d3532851 NSX 92012 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="INFO"] nsx_ujo.ncp.inventory Inventory send out update [('CREATE', {'external_id': '45c35fb8-f7bb-40b9-855b-9120aec7c9dc', 'display_name': 'test1', 'origin_properties': [], 'resource_type': 'ContainerIngressPolicy', 'container_project_id': 'd490cf06-46d3-4984-ab0d-13ca04de5a90', 'network_status': 'HEALTHY', 'network_errors': [], 'spec': 'rules:\n- host: FQDN\n http:\n paths:\n - backend:\n service:\n name: nginx-ping\n port:\n number: 80\n path: /\n pathType: Prefix\ntls:\n- secretName: test\n', 'container_cluster_id': '3d628019-de97-5257-a29a-eaf1c2b50d51', 'container_application_ids': ['4dd7b1e9-063c-4803-8e29-55776a92e13e'], 'tags': []})]

2024-07-11T07:41:58.307Z 944293c3-6624-47ef-8d97-9612d3532851 NSX 92012 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="INFO"] nsx_ujo.ncp.nsx.manager.nsxapi Imported certificate with tags [{'scope': 'ncp/version', 'tag': '1.2.0'}, {'scope': 'ncp/cluster', 'tag': 'pks-b593d4e0-56b0-47cf-9205-bdd9eea06b03'}, {'scope': 'ncp/project', 'tag': 'data'}, {'scope': 'k8s_resource_name','tag': 'test'}, {'scope': 'revision_number', 'tag': '7338456'}]

The certificate is present in NSX UI

Environment

Product version 1.17

Product version 1.18

Cause

Issue observed is related with the certificate format, even the certificate verification is successful the actual virtual server serving the ingress is failing to use the certificate.

Certificate format is in single line:

When certificate is converted into block format:

-----BEGIN CERTIFICATE-----
MIIJgjCCB2qgAwddIBAgITHgAAEFzZqyTr7rKWhQAAAAAQXDANBgkqhkiG9w0BAQsF
ADBMMRcwFddQYKCZImiZPyLGQBGRYHdndmcy1hZDETMBEGCgmSJomT8ixkARkWA3Z3
......... base64-encoded lines so that each line consists of exactly 64 characters except for the final line------
VdHWwuddl9NGD5aieNizvqBwwdSM8w+QkgQ0Z/jQB9Z74GKAEbMl4=
-----END CERTIFICATE-----

The ingress starts serving as expected

Resolution

Format the certificate using 64 character formatting then delete and recreate the secret to populate the new certificate.

Additional Information

This issue is possibly related to NSX related problem https://broadcomcms-software.wolkenservicedesk.com/external/article?articleNumber=318329