PCC HTTPS is not running

Products

• Services Suite

Issue/Introduction

Symptoms:

HTTPS service can be shown as not running if /var/tmp/commander.*.pid are missing.

[root@cdscil01 greenplum-cc]# service commander status
nodeagent is running
Jetty is running
httpd is running
Pivotal Command Center HTTPS is not running
Pivotal Command Center Background Worker is running
[root@cdscil01 greenplum-cc]#

Environment

Cause

Status of HTTPS service is checked using the pid files created for thin server processes under /var/tmp/. If these file are not available or have been deleted, you can see HTTPS status as not running even after restarting commander. If thin server processes are running, and pid files are not there, these process will lie orphan and will not be killed during restart.

Resolution

Step 1: Stop commander
[root@cdscil01 greenplum-cc]# service commander stop
Stopping PHD Manager Services...
Stopping Jetty [ OK ]
Stopping httpd: [ OK ]
Stopping Puppet [ OK ]
Pivotal Command Center HTTPS is not running
Stopping Pivotal Command Center Background Worker [ OK ]

Step 2: If thin server process is already running, you can create status pid files under /var/tmp/ . There must be 10 thin server process running at port 5500 to 5509 for normal operations.
Ex: Verify using netstat
$netstat -anp | egrep 550[0-9]
tcp 0 0 127.0.0.1:5500 0.0.0.0:* LISTEN 32371/thin server (
tcp 0 0 127.0.0.1:5501 0.0.0.0:* LISTEN 32382/thin server (
tcp 0 0 127.0.0.1:5502 0.0.0.0:* LISTEN 32393/thin server (
tcp 0 0 127.0.0.1:5503 0.0.0.0:* LISTEN 32404/thin server (
tcp 0 0 127.0.0.1:5504 0.0.0.0:* LISTEN 32417/thin server (
tcp 0 0 127.0.0.1:5505 0.0.0.0:* LISTEN 32429/thin server (
tcp 0 0 127.0.0.1:5506 0.0.0.0:* LISTEN 32442/thin server (
tcp 0 0 127.0.0.1:5507 0.0.0.0:* LISTEN 32456/thin server (
tcp 0 0 127.0.0.1:5508 0.0.0.0:* LISTEN 32472/thin server (
tcp 0 0 127.0.0.1:5509 0.0.0.0:* LISTEN 32503/thin server (

Create status pid files for all 10 processes.
Ex:
echo 32371 > /var/tmp/commander.5500.pid
$ls -ltr (Must have read / write permission for gpadmin user)
-rw-r--r-- 1 gpadmin gpadmin 5 May 28 20:40 commander.5500.pid

Or if the process are not responsive you can kill them and restart commander. 

$ps -ef | egrep "thin server"| egrep -v grep | awk '{print $2}' 
$kill <pid>