Port Conflicts on Pivotal Container Service (PKS) VM

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:

When UAA job has been bootstrapped before broker job, broker job on PKS node failed to start up. 

pivotal-container-service/2a08d81b-db83-48bb-8898-59432df05646:/var/vcap/bosh_ssh/bosh_758d42b2ee314ae# monit summary
The Monit daemon 5.2.5 uptime: 4d 14h 30m
Process 'pks-api' running
Process 'broker' not monitored
Process 'mariadb_ctrl' running
Process 'galera-healthcheck' running
Process 'gra-log-purger-executable' running
Process 'cluster_health_logger' running
Process 'uaa' running
Process 'bosh-dns' running
Process 'bosh-dns-healthcheck' running
System 'system_localhost' running

pivotal-container-service/2a08d81b-db83-48bb-8898-59432df05646:/var/vcap/bosh_ssh/bosh_758d42b2ee314ae# netstat -nap |grep 8080
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4734/java

pivotal-container-service/2a08d81b-db83-48bb-8898-59432df05646:/var/vcap/bosh_ssh/bosh_758d42b2ee314ae# monit start broker
pivotal-container-service/2a08d81b-db83-48bb-8898-59432df05646:/var/vcap/bosh_ssh/bosh_758d42b2ee314ae# monit summary
The Monit daemon 5.2.5 uptime: 4d 14h 31m
Process 'pks-api' running
Process 'broker' not monitored - start pending
Process 'mariadb_ctrl' running
Process 'galera-healthcheck' running
Process 'gra-log-purger-executable' running
Process 'cluster_health_logger' running
Process 'uaa' running
Process 'bosh-dns' running
Process 'bosh-dns-healthcheck' running
System 'system_localhost' running

pivotal-container-service/2a08d81b-db83-48bb-8898-59432df05646:/var/vcap/bosh_ssh/bosh_758d42b2ee314ae# monit summary
The Monit daemon 5.2.5 uptime: 4d 14h 32m
Process 'pks-api' running
Process 'broker' not monitored - start pending
Process 'mariadb_ctrl' running
Process 'galera-healthcheck' running
Process 'gra-log-purger-executable' running
Process 'cluster_health_logger' running
Process 'uaa' running
Process 'bosh-dns' running
Process 'bosh-dns-healthcheck' running
System 'system_localhost' running

pivotal-container-service/2a08d81b-db83-48bb-8898-59432df05646:/var/vcap/bosh_ssh/bosh_758d42b2ee314ae# monit summary
The Monit daemon 5.2.5 uptime: 4d 14h 32m
Process 'pks-api' running
Process 'broker' not monitored
Process 'mariadb_ctrl' running
Process 'galera-healthcheck' running
Process 'gra-log-purger-executable' running
Process 'cluster_health_logger' running
Process 'uaa' running
Process 'bosh-dns' running
Process 'bosh-dns-healthcheck' running
System 'system_localhost' running

Environment

Cause

UAA job started before the broker and occupied port 8080 which is also used by the broker.

pivotal-container-service/2a08d81b-db83-48bb-8898-59432df05646:/var/vcap/bosh_ssh/bosh_758d42b2ee314ae# netstat -nap |grep 8080
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 4734/java

Resolution

Here is the workaround:

1. monit stop uaa
2. monit start broker
3. monit start uaa
4. use "monit summary" to check availabilities