pks-nsx-t-osb-proxy fails to start with error tcp :3000: bind: address already in use

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:
BOSH shows that the PKS VM is in failing state

ubuntu@opsmanager-2-2:~$ bosh vms
Using environment '172.31.0.11' as client 'ops_manager'
Task 532
Task 533
Task 532 done
Task 533 done

Deployment 'pivotal-container-service-7fd9e00233c42f49846e'


Instance                                                        Process State  AZ   IPs          VM CID                                   VM Type  Active  
pivotal-container-service/f9ba432e-aba7-40de-b5eb-9b6b75b5d1cb  failing        az1  172.31.0.12  vm-d355afe9-2532-4f4a-952f-418507a600a8  large    true  

1 vms

From the monit summary on PKS vm pks-nsx-t-osb-proxy appears to in not monitored state

pivotal-container-service/f9ba432e-aba7-40de-b5eb-9b6b75b5d1cb:# monit summary
The Monit daemon 5.2.5 uptime: 9m 

Process 'pks-api'                   running
Process 'broker'                    running
Process 'pks-nsx-t-osb-proxy'       not monitored
Process 'mariadb_ctrl'              running
Process 'galera-healthcheck'        running
Process 'gra-log-purger-executable' running
Process 'cluster_health_logger'     running
Process 'uaa'                       running
Process 'telemetry'                 running
Process 'event-emitter'             running
Process 'bosh-dns'                  running
Process 'bosh-dns-resolvconf'       running
Process 'bosh-dns-healthcheck'      running
System 'system_localhost'           running

Logs on PKS vm under /var/vcap/sys/log/pks-nsx-t-osb-proxy/pks-nsx-t-osb-proxy.stderr.log report the following errors

2018/08/29 22:09:38 listen tcp :3000: bind: address already in use
2018/08/29 22:10:18 listen tcp :3000: bind: address already in use
2018/08/29 22:10:58 listen tcp :3000: bind: address already in use
2018/08/29 22:11:39 listen tcp :3000: bind: address already in use

Environment

Cause

Either there is a process running on PKS vm that is using port 3000 or the previous shutdown of pks-nsx-t-osb-proxy process didn't cleanup the os process. The offending process can be identified using

pivotal-container-service/f9ba432e-aba7-40de-b5eb-9b6b75b5d1cb:# netstat -lantup | grep 3000

tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      613/pks-nsx-t-osb-p

pivotal-container-service/f9ba432e-aba7-40de-b5eb-9b6b75b5d1cb:# ps -ef | grep 613

root       613     1  0 Aug22 ?        00:00:18 /var/vcap/packages/pks-nsx-t-osb-proxy/bin/pks-nsx-t-osb-proxy --config-file /var/vcap/jobs/pks-nsx-t-osb-proxy/config/config.json --listen-address=:3000

In this case the offending process was a stale pks-nsx-t-osb-proxy process. 

Resolution

Kill the offending process which should clear up port 3000 for usage. After killing the process restart pks-nsx-t-osb-proxy using monit.