Tanzu Packages Stuck in "Reconcile failed" State Due to TLS Validation Faults (x509: certificate signed by unknown authority) with Private OCI Registries
Article ID: 446807
Updated On:
Products
Issue/Introduction
Core addon packages (such as cert-manager, contour, harbor, or cluster-autoscaler-pkgi) deployed within a vSphere Kubernetes Service (VKS) workload cluster become trapped in a permanent Reconcile failed operational status.
When reviewing the cluster deployment profile via the Tanzu CLI, the package installation matrix displays partial framework initialization while dependent platform services fail to reconcile:
NAMESPACE NAME PACKAGE-NAME PACKAGE-VERSION STATUS
cert-manager cert-manager cert-manager.kubernetes.vmware.com 1.17.2+vmware.2-vks.1 Reconcile failed
tanzu-system-ingress contour contour.kubernetes.vmware.com 1.31.1+vmware.1-vks.1 Reconcile failed
tkg-system autoscaler-pkgi cluster-autoscaler.kubernetes.vmware.com 1.33.0+vmware.2-vks.1 Reconcile failed
vmware-system-tkg antrea antrea.tanzu.vmware.com 2.3.1+vmware.1-tkg.1 Reconcile succeeded
Additionally, query operations on the backing package repository resource surface an active fetch exception:
NAMESPACE NAME SOURCE STATUS
tkg-system private-registry (imgpkg) Reconcile failed: Fetching resources: Error (see .status.usefulErrorMessage for details)
When the Carvel management platform (kapp-controller) uses its internal imgpkg bundle fetching round-tripper to sync package manifests, it encounters a TLS validation fault. Examining the .status.usefulErrorMessage field of the failing resource exposes the root transport error:
Error while preparing a transport to talk with the registry:
Unable to create round tripper:
Get "https://<private-registry-fqdn>:<port>/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
Environment
vSphere Kubernetes Service (VKS)
Cause
This issue occurs because the cluster's package manager(kapp-controller) does not trust the external private OCI registry's SSL/TLS certificate. Because the private container registry uses a certificate signed by an internal corporate or self-signed Root Certificate Authority (CA), the cluster nodes reject the connection. The operating system's default trusted certificate stores inside the immutable VKS base images do not contain this custom public key. Consequently, the transport layer is aborted for security reasons, the repository manifest sync fails, and the dependent addon packages stall in a Reconcile failed loop.
Resolution
To restore successful repository synchronization, you must inject the base64-encoded public Root CA certificate chain that signs your private registry endpoint into the cluster's trust architecture. Choose one of the two standard implementation pathways below based on your configuration requirements.
Method A: Direct Injection at the PackageRepository Layer (Isolated Context)
This method encapsulates the custom Root CA public key inside an isolated Kubernetes secret mapped exclusively to the specific repository definition.
1. Save your private registry's public root certificate file (e.g., registry-root.crt) to your administrative terminal.
2. Generate a secure generic secret containing the public key within the target package synchronization namespace:
kubectl create secret generic registry-ca-bundle --from-file=ca.crt=registry-root.crt -n tkg-system
3. Open your declarative PackageRepository configuration manifest and inject the tls configuration block to reference the secret:
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
name: private-registry-repo
namespace: tkg-system
spec:
fetch:
imgpkgBundle:
image: <private-registry-fqdn>:<port>/tanzu/packages/standard/repo:v3.5.0
tls:
caBundleSecretRef:
name: registry-ca-bundle
4. Apply the updated layout to your cluster:
kubectl apply -f package-repository.yaml
Method B: Global Trust Topology Injection (Cluster-Wide Context)
For environments running modernized topology engine profiles (ClusterClass frameworks), you can distribute the trust fingerprint across every node in the cluster topology layer automatically.
- Convert your public Root CA certificate into a flat base64-encoded string block.
- Open your workload cluster deployment manifest (cluster.yaml).
- Inject the certificate block under the spec.topology.variables array targeting the trust configuration hook:
spec:
topology:
class: builtin-generic-v3.5.0 # Adjust to match your deployed ClusterClass version
version: v1.32.9+vmware.2
variables:
- name: trust
value:
additionalTrustedCAs:
- name: enterprise-registry-root-ca
data: |
-----BEGIN CERTIFICATE-----
MIIFlTCCBH2gAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwZzELMAkGA1UEBhMCVVMx
[Insert the complete base64 public certificate text block here]
-----END CERTIFICATE-----
4. Now, save the manifest and apply the structural adjustments to your management context:
kubectl apply -f cluster.yaml
The cluster controller plane will automatically initiate a safe, rolling lifecycle update across your node pools to install the new CA certificate across all control plane and worker nodes.
Verification
- Allow a few moments for the Carvel kapp-controller to re-trigger its synchronization loops against the newly trusted endpoint.
- Query the active package repository state to confirm that the fetch block has cleared:
tanzu package repository list -A
The status column should now transition to a permanent Reconcile succeeded posture.
- Verify that the dependent addon packages cleanly download their resource manifests and initialize successfully:
tanzu package installed list -A
All target platform packages will automatically resolve their dependencies and shift to Reconcile succeeded.
Feedback
