PKS api not able to login and receives an error "8443/oauth/token: dial tcp 10.229.2.4:8443: connect: connection refused"

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:

• Not able to login PKS api using uaa user
• monit summary on the PKS VM will fail with the following error.

/var/vcap/bosh/etc/monitrc:8: Warning: include files not found '/var/vcap/monit/job/*.monitrc'
The Monit daemon 5.2.5 uptime: 19m

• Attempting to redeploy PKS from Ops Manager will fail with the following error. 

Task 210029 

Task 210029 | 18:10:48 | Preparing deployment: Preparing deployment (00:00:07) 
Task 210029 | 18:11:08 | Preparing package compilation: Finding packages to compile (00:00:00) 
Task 210029 | 18:11:18 | Updating instance pivotal-container-service: pivotal-container-service/7190d127-09d2-46d3-a05d-77d025c36ccc (0) (canary) (00:01:03) 
L Error: Action Failed get_task: Task b40a20a8-6fbe-4747-4837-f144fe4dde51 result: 1 of 6 pre-start scripts failed. Failed Jobs: uaa. Successful Jobs: syslog_forwarder, bosh-dns, bosh-update-config, mysql, pks-api. 
Task 210029 | 18:12:21 | Error: Action Failed get_task: Task b40a20a8-6fbe-4747-4837-f144fe4dde51 result: 1 of 6 pre-start scripts failed. Failed Jobs: uaa. Successful Jobs: syslog_forwarder, bosh-dns, bosh-update-config, mysql, pks-api. 

Task 210029 Started Mon Feb 18 18:10:48 UTC 2019 
Task 210029 Finished Mon Feb 18 18:12:21 UTC 2019 
Task 210029 Duration 00:01:33 
Task 210029 error 

Capturing task '210029' output: 
Expected task '210029' to succeed but state is 'error' 

Exit code 1

Environment

VMware PKS 1.x

Resolution

Recreate the PKS VM using Ops manager via the following method.

• Select recreate all vms on the Bosh tile under director config.
• While applying changes select Bosh and PKS (without upgrade all cluster errand)