Loading PCC in Web browser failed with "Service Temporarily Unavailable"

Products

• Services Suite

Issue/Introduction

Symptoms:
When attempting to load PCC in web browser it failed with the error "Service Temporarily Unavailable" as shown in the screenshot below.

Environment

Cause

Pivotal Command Center HTTPS service ("thin server" processes") was not running which caused failure of loading PCC website.
 

RCA

1. Checked PCC service status and found HTTPS service was not running. And there wasn't any running "thin server" process in system

[root@mdw ~]# service commander status
nodeagent is running
Jetty is running
httpd is running
Pivotal Command Center HTTPS is not running
Pivotal Command Center Background Worker is running

[root@mdw ~]# ps -ef|grep thin
root 785490 744950 0 03:45 pts/43 00:00:00 grep thin

2. "thin server" logs indicated "No space left on device" error which caused "thin server" processes to exit abnormally

[root@mdw ~]# head /usr/local/pivotal-cc/log/thin.5501.log
/usr/local/pivotal-cc/vendor/bundle/ruby/1.9.1/gems/sequel-3.41.0/lib/sequel/adapters/postgres.rb:208: warning: Failed to set the default_internal encoding to UTF8: 'ERROR: failed to acquire resources on segment 0
'
/usr/local/pivotal-cc/vendor/bundle/ruby/1.9.1/gems/sequel-3.41.0/lib/sequel/adapters/postgres.rb:208: warning: Failed to set the default_internal encoding to UTF8: 'ERROR: failed to acquire resources on segment 0
'
log writing failed. No space left on device - /usr/local/pivotal-cc/log/production.log
log writing failed. No space left on device - /usr/local/pivotal-cc/log/production.log
log writing failed. No space left on device - /usr/local/pivotal-cc/log/production.log
......

3. Restart of PCC service could not solve the issue

4. Further check found some stale (empty) PID files for "thin server" process, which blocked new "thin server" processes from startingåÊup

[root@mdw ~]# ls -l /var/tmp
total 0
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5500.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5501.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5502.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5503.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5504.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5505.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5506.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5507.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5508.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5509.pid

5. As mentioned in this article "Pivotal Command Center HTTPS is not running" will be shown when there is not enough "thin server" processes up and running

Resolution

1. Check "thin server" logs for any errors and fix them. In this case, do some cleanup toåÊfree more disk space in root partition (/)

2. Remove any stale PID files for "thin server" process before start commander service