pks-api pre-start script failing with error "No certificate matches private key"

Products

• VMware
• VMware vSphere with Tanzu

Issue/Introduction

Symptoms:

TKGi PKS-API pivotal-container-service VM is in a failing status where pre-start script is failing to run with the error "No certificate matches private key"

 

Instance                                                        Process State  AZ   IPs          VM CID                                   VM Type     Active  Stemcell  
pivotal-container-service/ca6b9b5b-242f-41f0-b6f6-f7bcee6d07c0  stopped        az1  172.30.0.13  vm-fab6624b-2790-4f62-bf34-d9ec361b1555  large.disk  true    bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.296  
pks-db/1f84e47b-22d3-4c12-9532-01257e3cc917                     running        az1  172.30.0.12  vm-b54503c5-ef4c-47f4-a29c-658cafa4dd17  large.disk  true    bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.296  

ubuntu@opsmgr-01-slot-20-pez-vmware-com:~$ bosh -d pivotal-container-service-9691ed10993ec1e3a4f9 ssh pivotal-container-service/ca6b9b5b-242f-41f0-b6f6-f7bcee6d07c0
Using environment '172.30.0.11' as user 'director'

pivotal-container-service/ca6b9b5b-242f-41f0-b6f6-f7bcee6d07c0:~# sudo /var/vcap/jobs/pks-api/bin/pre-start
Setting up key store, trust store and installing certs.
Importing keystore pks_api_tls_cert.p12 to /var/vcap/jobs/pks-api/config/keystore.jks...
Importing keystore pks_api_internal_tls_cert.p12 to /var/vcap/jobs/pks-api/config/keystore.jks...
Importing keystore pks_db_tls_cert.p12 to /var/vcap/jobs/pks-api/config/db-keystore.jks...
No certificate matches private key

Error from the changelog in OpsManager

Task 68 | 14:28:58 | L executing pre-start: pivotal-container-service/ca6b9b5b-242f-41f0-b6f6-f7bcee6d07c0 (0) (canary) (00:00:31) L Error: Action Failed get_task: Task f15356d1-12e9-4da3-6ed5-2ee198a125dc result: 1 of 6 pre-start scripts failed. Failed Jobs: pks-api. Successful Jobs: bpm, bosh-update-config, bosh-dns, syslog_forwarder, uaa. 

Task 68 | 14:29:02 | Error: Action Failed get_task: Task f15356d1-12e9-4da3-6ed5-2ee198a125dc result: 1 of 6 pre-start scripts failed. Failed Jobs: pks-api. Successful Jobs: bpm, bosh-update-config, bosh-dns, syslog_forwarder, uaa.

 

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x
Tanzu Kubernetes Grid Integrated Edition 1.1.14.1

Cause

PKS-API is complaining the private key does not match for the certificate for nsx_spueruser.crt

Resolution

Bosh SSH into the PKS-API pivotal-container-service VM and run those commands to check if the keys match

openssl pkey -in /var/vcap/jobs/pks-api/config/nsx_superuser_private.key -pubout -outform pem | sha256sum
openssl x509 -in /var/vcap/jobs/pks-api/config/nsx_superuser.crt -pubkey -noout -outform pem | sha256sum

if the keys do not match, follow this KB to renew the nsx-t-superuser-certificate used by Principal Identity user https://kb.vmware.com/s/article/80355