Creating ingress in TKGI with NSX-T does not send corresponding certificate in NSX Load Balancer termination point
Article ID: 298652
Updated On:
Products
VMware Tanzu Kubernetes Grid Integrated Edition
VMware Tanzu Kubernetes Grid Integrated Edition (Core)
VMware Tanzu Kubernetes Grid Integrated (TKGi)
VMware Tanzu Kubernetes Grid Integrated Edition 1.x
VMware Tanzu Kubernetes Grid Integrated EditionStarter Pack (Core)
Issue/Introduction
When you try to verify the internal certificate, you get the NSX certificate instead and it is defined in secret tls:
Ingress example:
Ingress is created successfully, however the certificate is not visible in NSX Load balancer termination.
openssl s_client -showcerts -CAfile FWCARoot.cer -connect <FQDN>:443 CONNECTED(00000003) depth=0 CN = nsx-lb verify error:num=18:self signed certificate verify return:1 depth=0 CN = nsx-lb verify return:1
Ingress example:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-test namespace: ingress-test spec: rules: - host: <FQDN> http: paths: - backend: serviceName: tm-svc servicePort: 80 tls: - secretName: <FQDNSecret>
Ingress is created successfully, however the certificate is not visible in NSX Load balancer termination.
Environment
Product Version: 1.17.x
Resolution
The following check can be completed to verify public key, private key, and validity matches:
Verify the private and public keys match the PEM validity, cert is readible, and use openssl to confirm. However when ingress is defined with the secret with the certificate below it is not imported as expected. To confirm this (I have a single master), I have logged to the master node:
Then delete and re-create the ingress. Here are some of the messages I have found in the logs:
Please note the error comes from the vmware_nsxlib.v3.client I have further analyzed and tested with v3 certificate with the following extension file:
And my v3 cert was imported successfully. Next I have compared both certs. And here is the difference in the extensions:
Verify the private and public keys match the PEM validity, cert is readible, and use openssl to confirm. However when ingress is defined with the secret with the certificate below it is not imported as expected. To confirm this (I have a single master), I have logged to the master node:
tail -f /var/vcap/sys/log/ncp/ncp.stdout.log
Then delete and re-create the ingress. Here are some of the messages I have found in the logs:
[nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] vmware_nsxlib.v3.client The HTTP request returned error code 400, whereas 201/200 response codes were expected. Response body {'module_name': 'internal-framework', 'httpStatus': 'BAD_REQUEST', 'error_code': 2002, 'error_message': 'Invalid PEM data received for certificate.'}
[nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="ERROR" errorCode="NCP00065"] nsx_ujo.ncp.nsx.manager.nsxapi Failed to import certificate with tags [{'tag': '1.2.0', 'scope': 'ncp/version'}, {'tag': 'pks-eae6d0c8-0491-4dad-9848-7572b06ce250', 'scope': 'ncp/cluster'}, {'tag': 'default', 'scope': 'ncp/project'}, {'tag': 'tea-secret-nsx-t', 'scope': 'k8s_resource_name'}]: Unexpected error from backend manager (['sc3-nsxmgr-vip.slot-50.pez.vmware.com']) for POST api/v1/trust-management/certificates?action=import: Invalid PEM data received for certificate.
Please note the error comes from the vmware_nsxlib.v3.client I have further analyzed and tested with v3 certificate with the following extension file:
cat v3.ext authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
And my v3 cert was imported successfully. Next I have compared both certs. And here is the difference in the extensions:
X509v3 extensions: X509v3 Authority Key Identifier: keyid:99:D3:80:33:92:1F:64:A1:43:AD:BE:6D:AF:39:57:68:CE:13:23:78 X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
When compared to the following:
X509v3 extensions: X509v3 Subject Key Identifier: 6A:2C:74:C9:F7:5C:5B:EA:F7:90:1F:CD:C5:C8:D4:E5:75:21:56:C1 X509v3 Authority Key Identifier: keyid:B4:F1:48:79:DC:D8:25:23:86:A3:25:1B:4B:95:41:54:25:2A:22:FF X509v3 CRL Distribution Points: Full Name: URI:http://pki.<provider>/<provider>.crl Authority Information Access: CA Issuers - URI:http://pki.<provider>/<provider>.crt X509v3 Key Usage: critical Digital Signature, Key Encipherment 1.3.6.1.4.1.311.21.7: 0/.'+.....7.............'...'...j............d... X509v3 Extended Key Usage: TLS Web Server Authentication 1.3.6.1.4.1.311.21.10: 0.0 ..+.......
I am suspecting there is a value in the v3 extensions that the system is not capable of reading/importing is the cause of the issue and respectively ncp fails to import the cert to NSX-T. Reducing the extension details solved the problem and the certificate was visible in NSX-T and working for connections.
Feedback
Yes
No
Powered by
