How to identify the certificate used when Ingress is not using the new certificate applied in Tanzu Kubernetes Grid Integrated Edition
Article ID: 298670
Updated On:
Products
VMware Tanzu Kubernetes Grid Integrated Edition
Issue/Introduction
When ingress is created with a TLS secret in a Tanzu Kubernetes Grid Integrated Edition (TKGI) cluster with NSX-T, in the background on NSX, several objects are created (NSX load balancer, certificate etc.).
A common practice is to use a wildcard in the certificate. For example: CN=*.example.com
As multiple teams create their own ingress, they also specify their own certificate and its possible that multiple ingress's are created with different certificates all matching the same wildcard (CN=*.example.com).
When a user is performing a certificate rotation in their ingress. The certificate is valid and has been applied as a new secret, but no update is showing.
The secret is showing the new certificate, both the ingress and pod were redeployed, yet the old certificate is appearing on the browser instead.
The reason for this behavior is within NSX manager and how the certificates are handled. if there is a secret that contains the old certificate, and there is also secret that contains the new certificate, it is possible that the old certificate will be present on NSX load balancer instead of the new.
To verify this, run curl against the ingress and verify certificate validity:
curl -k -v https://host.example.com
Another method is to use openssl:
openssl s_client -showcerts -connect host.example.com:443 -servername host.example.com
A common practice is to use a wildcard in the certificate. For example: CN=*.example.com
As multiple teams create their own ingress, they also specify their own certificate and its possible that multiple ingress's are created with different certificates all matching the same wildcard (CN=*.example.com).
When a user is performing a certificate rotation in their ingress. The certificate is valid and has been applied as a new secret, but no update is showing.
The secret is showing the new certificate, both the ingress and pod were redeployed, yet the old certificate is appearing on the browser instead.
The reason for this behavior is within NSX manager and how the certificates are handled. if there is a secret that contains the old certificate, and there is also secret that contains the new certificate, it is possible that the old certificate will be present on NSX load balancer instead of the new.
To verify this, run curl against the ingress and verify certificate validity:
curl -k -v https://host.example.com
Another method is to use openssl:
openssl s_client -showcerts -connect host.example.com:443 -servername host.example.com
Environment
Product Version: 1.10
Resolution
From NSX UI, is not easy to find out from which secret the certificate is created because only the validity CN and a few other details are visible. To get the full details, a curl command has to be executed.
Run the following commands from a master on the cluster to get more certificate details:
1. SSH to one of the masters:
bosh -d <service_instance> ssh master/0
2. Run this command: sudo -i
3. Get the top lines from: head -n20 /var/vcap/jobs/pks-nsx-t-prepare-master-vm/bin/pre-start
4. Copy and paste these lines in the console, they create environment variables that will be used in the curl command below.
5. Get Certificate details with this curl command:
curl -X GET "https://${NSX_MANAGER_HOST}/api/v1/trust-management/certificates/5801641d-efa9-4f21-bfef-fd8e5f7cdfd0" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --key ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE} -H "accept: application/json" -H "Content-Type: application/json"
Where 5801641d-efa9-4f21-bfef-fd8e5f7cdfd0 is the certificate ID from the NSX UI. The output would look similar to the following:
Looking at the tags, you can identify this certificate was created in the default namespace and secret tea-secret-nsx-t are the origin of this certificate.
By deleting the secret and re-creating the secret with the new certificate, the system has one consistent certificate in place.
Run the following commands from a master on the cluster to get more certificate details:
1. SSH to one of the masters:
bosh -d <service_instance> ssh master/0
2. Run this command: sudo -i
3. Get the top lines from: head -n20 /var/vcap/jobs/pks-nsx-t-prepare-master-vm/bin/pre-start
#!/bin/bash set -e NSX_MANAGER_CA_CERT_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_ca.crt" NSX_MANAGER_CLIENT_CERT_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_client.crt" NSX_MANAGER_CLIENT_KEY_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_client.key" NSX_MANAGER_SUPERUSER_CERT_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_superuser.crt" NSX_MANAGER_SUPERUSER_KEY_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_superuser.key" export PATH="/var/vcap/packages/pks-nsx-t-jq/bin/:/var/vcap/packages/pks-nsx-t-curl/bin/:$PATH" export NSX_MANAGER_HOST='NSXhostname' export NSX_MANAGER_IP='NSXIP' export NSX_MANAGER_INSECURE='false' export NSX_MANAGER_CLIENT_CERTIFICATE="$NSX_MANAGER_CLIENT_CERT_FILE" export NSX_MANAGER_CLIENT_KEY="$NSX_MANAGER_CLIENT_KEY_FILE" export NSX_MANAGER_CA_CERT="$NSX_MANAGER_CA_CERT_FILE"
4. Copy and paste these lines in the console, they create environment variables that will be used in the curl command below.
5. Get Certificate details with this curl command:
curl -X GET "https://${NSX_MANAGER_HOST}/api/v1/trust-management/certificates/5801641d-efa9-4f21-bfef-fd8e5f7cdfd0" --cert ${NSX_MANAGER_CLIENT_CERT_FILE} --key ${NSX_MANAGER_CLIENT_KEY_FILE} --cacert ${NSX_MANAGER_CA_CERT_FILE} -H "accept: application/json" -H "Content-Type: application/json"
Where 5801641d-efa9-4f21-bfef-fd8e5f7cdfd0 is the certificate ID from the NSX UI. The output would look similar to the following:
{
"pem_encoded" : "-----BEGIN CERTIFICATE-----
<----OMITTED---->
-----END CERTIFICATE-----\n",
"used_by" : [ ],
"resource_type" : "certificate_signed",
"id" : "5801641d-efa9-4f21-bfef-fd8e5f7cdfd0",
"display_name" : "5801641d-efa9-4f21-bfef-fd8e5f7cdfd0",
"tags" : [ {
"scope" : "ncp/version",
"tag" : "1.2.0"
}, {
"scope" : "ncp/cluster",
"tag" : "pks-32114aa0-9d37-4a76-95cd-e94dbc2335b1"
}, {
"scope" : "ncp/project",
"tag" : "default"
}, {
"scope" : "k8s_resource_name",
"tag" : "tea-secret-nsx-t"
} ],
"_create_user" : "pks-32114aa0-9d37-4a76-95cd-e94dbc2335b1",
"_create_time" : 1625606238512,
"_last_modified_user" : "pks-32114aa0-9d37-4a76-95cd-e94dbc2335b1",
"_last_modified_time" : 1625606238512,
"_system_owned" : false,
"_protection" : "NOT_PROTECTED",
"_revision" : 0
}
Looking at the tags, you can identify this certificate was created in the default namespace and secret tea-secret-nsx-t are the origin of this certificate.
By deleting the secret and re-creating the secret with the new certificate, the system has one consistent certificate in place.
Feedback
Yes
No
Powered by
