Error: "failed to call extension handler 'generate-patches.runtime-extension'" due to harbor-tls secret when creating TKG clusters with the error "invalid certificate: found labeled non-certificate secret"
search cancel

Error: "failed to call extension handler 'generate-patches.runtime-extension'" due to harbor-tls secret when creating TKG clusters with the error "invalid certificate: found labeled non-certificate secret"

book

Article ID: 441275

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

Kubernetes workload cluster creation fails from Aria Automation (VCF Automate) or the Local Consumption interface.

The workload cluster status reports: TopologyReconciled: error computing the desired state of the Cluster topology: failed to apply patches: failed to generate patches for patch "default": failed to call extension handler "generate-patches.runtime-extension": got failure response, please check controller logs for errors

Checking the runtime-extension pod logs in the vmware-system-tkg namespace reveals the following error:

kubectl logs -n <TKG-POD-NAME> <RUNTIME-EXTENSTION-POD-NAME>

"error during patch generation" err="unable to retrieve default registry secret: error decoding PEM block: invalid certificate: found labeled non-certificate secret harbor-tls in namespace svc-harbor-domain-c1016"

Environment

VMware Cloud Foundation (VCF) 9.0
vSphere with Tanzu 9.0
Harbor Supervisor Service

Cause

The Cluster API generate-patches.runtime-extension webhook fails to inject the default Harbor container registry certificate into the new workload cluster.

This occurs because the harbor-tls secret in the svc-harbor-domain-c1016 namespace is missing the ca.crt data. Within VKS, if a secret is labeled with "managed-by": "vmware-vRegistry" and does not possess the cert-manager.io/common-name annotation, the internal logic strictly expects the secret to contain a Certificate Authority (CA) certificate.

If the Harbor Supervisor Service YAML configuration was applied with only the tls.crt and tls.key fields—omitting the ca.crt field—the VKS parser fails to decode the expected PEM block and rejects the secret as an "invalid certificate."

Resolution

The Harbor Supervisor Service configuration must be updated natively through the vCenter UI to include the ca.crt field, allowing the runtime extension to fetch the required CA data. 

  1. Obtain the ca.crt from your environment and save to a .crt or .cer file and obtain the contents. You can verify the current secrets for the harbor with below commands. It is likely that ca.crt is missing or malformed in this condition:

    kubectl get secret harbor-tls -n svc-harbor-domain-<ID> -o jsonpath='{.data.tls\.crt}' | base64 -d
    kubectl get secret harbor-tls -n svc-harbor-domain-<ID> -o jsonpath='{.data.tls\.key}' | base64 -d
    kubectl get secret harbor-tls -n svc-harbor-domain-<ID> -o jsonpath='{.data.ca\.crt}' | base64 -d

    Use OpenSSL to verify that the certificate is readable and check its attributes (such as validity dates and issuer):

    openssl x509 -in ca.crt -text -noout
    openssl x509 -in tls.crt -text -noout

    Etract and compare the modulus of both the certificate and the key. The resulting MD5 hashes must be exactly identical.

    For the certificate:

    openssl x509 -noout -modulus -in tls.crt | openssl md5

    For the private key:

    openssl rsa -noout -modulus -in tls.key | openssl md5

    Verify the TLS Certificate is signed by the CA Certificate Use the CA certificate file to verify the TLS certificate's chain of trust.
    Note: If there are issues verifying the cert, ensure that any issues with it are resolved (wrong format, extra spaces, characters etc.) are resolved before adding it to the harbor.

     

    openssl verify -CAfile ca.crt tls.crt

  2. Log in to the vSphere Client.

  3. Navigate to Workload Management > Services (or Supervisor Management > Services).

  4. Select the Harbor service and click Actions > Manage Service.

  5. Backup the current yaml contents in case of wrong/unintended modifications. In the YAML Service Config window, locate the tlsCertificate: block.

  6. Add and properly populate the ca.crt field alongside the existing tls.crt and tls.key. Ensure strict YAML indentation is maintained:

    tlsCertificate:
  tlsSecretLabels: 
    managed-by: vmware-vRegistry
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    <CA_CERT_DATA>
    -----END CERTIFICATE-----
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    <TLS_CERT_DATA>
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    <TLS_KEY_DATA>
    -----END PRIVATE KEY-----

  7. Click Finish to apply the configuration.

  8. The Harbor Supervisor Service will automatically reconcile and recreate the harbor-tls secret with the correct CA certificate data. Subsequent Kubernetes cluster deployments will successfully pass the topology reconciliation phase. 

  9. You will start to observe tasks in vCenter as part of cluster privisoning.

Additional Information

To quickly extract the backend runtime-extension logs to verify if this article applies to your environment, execute the following commands on the Supervisor Control Plane:

  1. Retrieve the pod name: kubectl get pods -n vmware-system-tkg | grep runtime-extension

  2. Extract the error: kubectl logs -n vmware-system-tkg <POD_NAME> | grep "PEM block: invalid certificate: found labeled non-certificate secret harbor-tls"

Powered by Wolken Software