Pinniped post-deploy job stuck in Error status due to "x509 : certificate signed by unknown authority (possibly because of “crypto: rsa : verification error)"

Products

• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid Management
• VMware Tanzu Kubernetes Grid Plus

Issue/Introduction

• You will observe that the pinniped-post-deploy-job-* in the pinniped-supervisor namespace is in Error status as shown in the output below  :

kubectl get pods -n pinniped-supervisor

NAMESPACE            NAME                                   READY   STATUS 
pinniped-supervisor  pinniped-post-deploy-job-xxxxxxxxx.     0/1     Error  

 

• Upon describing the pinniped-post-deploy-job-* ,  you will get below error :

Error : Could not complete pinniped login  : could not perform OIDC discovery for “https://x.x.x.x”  : Get “https://x.x.x.x/.well-known/openid-configuration” : x509 : certificate signed by unknown authority (possibly because of “crypto: rsa : verification error” while trying to verify candidate authority certificate “tkg-pinniped” 

Environment

VMware Tanzu Kubernetes Grid 1.x

VMware Tanzu Kubernetes Grid 2.x

Resolution

• You can fix this problem by deleting the pinniped-post-deploy-job under the pinniped-supervisor namespace . 
   

kubectl get app -n tkg-system pinniped
NAME       DESCRIPTION                                  SINCE-DEPLOY   AGE
pinniped   Reconcile failed: Deploying: exit status 1   10m43s          69m

kubectl delete jobs.batch -n pinniped-supervisor pinniped-post-deploy-job
job.batch "pinniped-post-deploy-job" deleted

 

• The app object reconciliation is done by the kapp-controller every 5 minutes so you may have to wait for some time before the reconciliation kicks off. Once the app has started reconciling you should see the following output :

 

kubectl get app -n tkg-system pinniped
NAME       DESCRIPTION   SINCE-DEPLOY   AGE
pinniped   Reconciling   3s             69m

kubectl get app -n tkg-system pinniped
NAME       DESCRIPTION           SINCE-DEPLOY   AGE
pinniped   Reconcile succeeded   37s            70m

 

kubectl get jobs -n pinniped-supervisor
NAME                                              COMPLETIONS   DURATION   AGE
pinniped-post-deploy-job-xxxxxx   1/1                           10s         92s