TKGi NCP error - "Secret xxx-secret with the same PEM data has been imported, use a different certificate with the secret instead"
search cancel

TKGi NCP error - "Secret xxx-secret with the same PEM data has been imported, use a different certificate with the secret instead"

book

Article ID: 378128

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Integrated Edition VMware NSX

Issue/Introduction

Updating an Ingress secret to renew the TLS certificate fails with an error that the secret has the same PEM data that has been imported.

Creating a new ingress and secret can also fail with this error if the same certificate with same PEM data is already existing in NSX.

NCP logs show:

2025-12-16T17:02:57.406Z 9a4875ce-b370-4d1b-8cbb-69abbe0a0543 NSX 11022 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] nsx_ujo.ncp.nsx.lb_l7_service Secret example-web2-ingress-secret with the same PEM data has been imported, use a different secret instead

Environment

TKGi 1.20.x
NSX-T 4.1.2.1

Cause

The certificate with the same PEM data was already imported in NSX Manager.

Resolution

Find the k8s secret that corresponds to the certificate that was already imported in NSX Manager.

1. Get the thumbprint of the cert from the secret (example-web2-ingress-secret as seen in the error log).

$ kubectl -n web2 get secret example-web2-ingress-secret -o jsonpath="{.data.tls\.crt}" | base64 --decode > /tmp/web2.pem

$ openssl x509 -noout -fingerprint -sha256 -in /tmp/web2.pem
sha256 Fingerprint=XX:YY:ZZ:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:XX:YY:ZZ

 

2. SSH into a Master VM and get all certificates that are in the NSX Manager and save them in a text file.

# set the following 2 envars to the NSX Manager FQDN
export NSX_MANAGER_HOST='nsxmgr.example.com'
export NSX_MANAGER_IP='nsxmgr.example.com'

# export the following as they are
export NSX_MANAGER_CA_CERT_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_ca.crt"
export NSX_MANAGER_CLIENT_CERT_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_client.crt"
export NSX_MANAGER_CLIENT_KEY_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_client.key"
export NSX_MANAGER_SUPERUSER_CERT_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_superuser.crt"
export NSX_MANAGER_SUPERUSER_KEY_FILE="/var/vcap/jobs/pks-nsx-t-prepare-master-vm/config/nsx_t_superuser.key"
export PATH="/var/vcap/packages/pks-nsx-t-jq/bin/:/var/vcap/packages/pks-nsx-t-curl/bin/:$PATH"
export NSX_MANAGER_INSECURE='false'

# run curl to hit the certificates endpoint
curl -v -E "${NSX_MANAGER_SUPERUSER_CERT_FILE}"  --key "${NSX_MANAGER_SUPERUSER_KEY_FILE}" --cacert "${NSX_MANAGER_CA_CERT_FILE}" https://$NSX_MANAGER_HOST/api/v1/trust-management/certificates > /tmp/certs.txt

 

3. From the saved list of certs retrieved from the NSX Manager, check if the thumbprint is included:

$ grep -C25 'XX:YY:ZZ:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:XX:YY:ZZ' /tmp/certs.txt
...
  }, {
    "pem_encoded" : "-----BEGIN CERTIFICATE-----\nMIIF0zCCA7ugAwI...==\n-----END CERTIFICATE-----\n",
    "has_private_key" : true,
    "used_by" : [ ],
    "leaf_certificate_sha_256_thumbprint" : "XX:YY:ZZ:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:XX:YY:ZZ",
    "resource_type" : "certificate_self_signed",
    "id" : "xxx-c13b-4f46-a367-e0a814c1dcd4",
    "display_name" : "xxx-c13b-4f46-a367-e0a814c1dcd4",
    "tags" : [ {
      "scope" : "ncp/version",
      "tag" : "1.2.0"
    }, {
      "scope" : "ncp/cluster",
      "tag" : "pks-xxx-72af-419a-91b3-373b8cc5bceb"
    }, {
      "scope" : "ncp/project",
      "tag" : "nginx"
    }, {
      "scope" : "k8s_resource_name",
      "tag" : "example-ingress-secret"
    }, {
      "scope" : "revision_number",
      "tag" : "2906851"
    } ],
    "_create_time" : 1765902025594,
    "_create_user" : "pks-xxx-72af-419a-91b3-373b8cc5bceb",
    "_last_modified_time" : 1765902025594,
    "_last_modified_user" : "pks-xxx-72af-419a-91b3-373b8cc5bceb",
    "_system_owned" : false,
    "_protection" : "REQUIRE_OVERRIDE",

 

4. From the output above, get the "ncp/cluster", "ncp/project" and "k8s_resource_name" tags.  These are the cluster ID, namespace and secret name respectively.  This helps identify what is the particular secret that corresponds to the certificate that is actually saved in NSX Manager.

$ kubectl -n nginx get secret example-ingress-secret
NAME                     TYPE                DATA   AGE
example-ingress-secret   kubernetes.io/tls   2      3d1h

 

5. Assess and decide if the secret should be deleted or not.

 

In case where a few teams use a common wildcard certificate (e.g., Subject: CN=*.example.com), it can also be useful to check by CN.  The following command will examine each certificate retrieved from NSX (Step 2) and output the "Subject" information, which you can then grep for the wildcard CN to confirm if a certificate for it is already existing in NSX.

$ grep pem_encoded /tmp/certs.txt | awk -F: '{print $2}' | while read -r pem; 
do
     clean_pem="${pem//\"/}";
     printf "%b" "$clean_pem";
     printf "%b" "$clean_pem" | openssl x509 -text -noout | grep "Subject:";
     echo "########################"; 
done

If you see the wildcard CN in question, you can then find the corresponding certificate item in the saved file (/tmp/certs.txt in the above example) and get the cluster ID, namespace and secret name.  Once those information is available, assess and decide if the secret should be deleted or not.

 

Powered by Wolken Software