pinniped-post-deploy-job Fails with Error: secrets "None" not found"

Products

Tanzu Kubernetes Runtime

Issue/Introduction

You will notice that "pinniped-post-deploy-job" is continuously failing:

ubuntu@jumpbox:~$ kubectl get po -n pinniped-supervisor
NAME                                               READY   STATUS    RESTARTS   AGE
pinniped-post-deploy-controller-5d854989cf-bc2cz   1/1     Running   0          52d
pinniped-post-deploy-job-6z9mx                     0/1     Error     0          3m46s
pinniped-post-deploy-job-8lczp                     0/1     Error     0          7m13s
pinniped-post-deploy-job-fzhbx                     0/1     Error     0          7m19s
pinniped-post-deploy-job-hwbc5                     0/1     Error     0          6m49s
pinniped-post-deploy-job-lhdb5                     0/1     Error     0          7m26s
pinniped-post-deploy-job-pfbqg                     0/1     Error     0          7m16s
pinniped-post-deploy-job-twpnb                     0/1     Error     0          7m23s

When you examine the logs for the failed job, you will see:

2025-05-26T14:55:19.144Z	INFO	configure/configure.go:155	Override certificate with user provided secret None
2025-05-26T14:55:19.146Z	ERROR	configure/configure.go:157	secrets "None" not found

When this job fails, it prevents Pinniped from automatically renewing its certificates. As a result, once the certificates expire, they will not be rotated automatically, potentially causing authentication issues.

Environment

TKGm 2.4.0+

Cause

This article addresses one possible cause of the issue: a misconfigured pinniped-package secret. Specifically, the problem can occur when the custom_cluster_issuer and custom_tls_secret fields in the secret are missing quotation marks.

Note: This configuration is stored in the <management-cluster-name>-pinniped-package secret in the tkg-system namespace. To inspect the values.yaml section, you will need to decode it using base64 -d (echo "base64 string" | base6 -d).

Resolution

Extract the the "values.yaml" section from the secret:

kubectl get secret CLUSTER-NAME-pinniped-package -n tkg-system -o jsonpath="{.data.values\.yaml}" | base64 -d > FILENAME.yaml

Edit the generated file and ensure that the custom_cluster_issuer and custom_tls_secret fields are enclosed in quotation marks. Below is an example of how these fields should appear in a working configuration:

infrastructure_provider: vsphere
tkg_cluster_role: management
custom_cluster_issuer: ""
custom_tls_secret: ""

NOT

infrastructure_provider: vsphere
tkg_cluster_role: management
custom_cluster_issuer:
custom_tls_secret:

Once you've updated the file, encode it back to base64 using the following command. Replace FILENAME.yaml with the name of your modified file:

cat FILENAME.yaml | base64 -w 0 > values.yaml

Copy the base64-encoded output and replace the values.yaml section in the secret with it. You can do this by editing the secret directly:

kubectl edit secret CLUSTER-NAME-pinniped-package -n tkg-system

apiVersion: v1
data:
  values.yaml: <PASTE HERE>

After the the pinniped package reconciles, you will see that the pinniped-post-deploy-job succeeds again:

ubuntu@jumpbox:~$ kubectl get jobs -n pinniped-supervisor pinniped-post-deploy-job
NAME                       COMPLETIONS   DURATION   AGE
pinniped-post-deploy-job   1/1           7s         40s