Applying changes to Tanzu Kubernetes Grid Integrated Edition fails with "1 of 6 pre-start scripts failed. Failed Jobs: pks-api"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:

• When applying changes, you see messages similar to the following in the bosh task output:

Task 195146 | 00:43:26 | Updating instance pivotal-container-service: pivotal-container-service/98ff3dcc-bf89-4ea1-b677-da03cdacded5 (0) (canary)
Task 195146 | 00:43:29 | L executing pre-stop: pivotal-container-service/1b08d085-56e8-483e-86ba-d023d0c06744 (0) (canary)
Task 195146 | 00:43:30 | L executing drain: pivotal-container-service/1b08d085-56e8-483e-86ba-d023d0c06744 (0) (canary)
Task 195146 | 00:43:31 | L stopping jobs: pivotal-container-service/1b08d085-56e8-483e-86ba-d023d0c06744 (0) (canary)
Task 195146 | 00:43:32 | L executing post-stop: pivotal-container-service/1b08d085-56e8-483e-86ba-d023d0c06744 (0) (canary)
Task 195146 | 00:43:49 | L installing packages: pivotal-container-service/1b08d085-56e8-483e-86ba-d023d0c06744 (0) (canary)
Task 195146 | 00:43:53 | L configuring jobs: pivotal-container-service/1b08d085-56e8-483e-86ba-d023d0c06744 (0) (canary)
Task 195146 | 00:43:53 | L executing pre-start: pivotal-container-service/98ff3dcc-bf89-4ea1-b677-da03cdacded5 (0) (canary) (00:00:35)
                      L Error: Action Failed get_task: Task cd03aad8-0ac8-466f-5d43-b415e56cf220 result: 1 of 6 pre-start scripts failed. Failed Jobs: pks-api. Successful Jobs: bpm, syslog_forwarder, bosh-dns, bosh-update-config, uaa.
Task 195146 | 00:44:01 | Error: Action Failed get_task: Task cd03aad8-0ac8-466f-5d43-b415e56cf220 result: 1 of 6 pre-start scripts failed. Failed Jobs: pks-api. Successful Jobs: bpm, syslog_forwarder, bosh-dns, bosh-update-config, uaa.​​​​​​​

• You see messages related to a keytool error about the certificate not being X.509 in the pks-api/pre-start.stdout.log, similar to the following:

Setting up key store, trust store and installing certs.
keytool error: java.lang.Exception: Input not an X.509 certificate
pre-start.stdout.log 

Cause

Either a certificate is misformatted or there is not a certificate present on the TKGI Tile > Networking > NSX Manager CA.

Resolution

 

1. Connect to the TKGI API VM. Note: You can do this with the bosh -d <deployment-id> ssh pivotal-container-service/0 command.
2. Check the nsx-t ca certificate with the following command: cat /var/vcap/jobs/pks-api/config/nsx_manager_ca.crt. If the output states "none" then this is the problem certificate. Otherwise, review all the certificates in the TKGI tile ensuring all of them are in X.509 pem format.
3. Get the NSX-T CA certificate. This can be obtained by viewing the certificate when the browser is connected to the NSX-T Manager UI via the API VIP.
4. Copy the NSX-T certificate to the TKGI Tile > Networking > NSX Manager CA.
5. Attempt to apply changes again.