GPCC unable to start - Failed to detect agent start status

Products

• VMware Tanzu Greenplum

Issue/Introduction

- gpcc 4.11.1
- Unable to start the gpcc due to an error:

Failed to detect agent start status

Environment

Product Version: 5.27

Resolution

- upon checking the webserver log we found the information that port is in use
 

grep CRITICAL ./webserver.*
./webserver.2020-12-18_145351.log:2020-12-18 13:30:54 [CRITICAL]Can't listen on port "28080": listen tcp :28080: bind: address already in use
./webserver.2020-12-18_145351.log:2020-12-18 13:30:54 [CRITICAL]Can't listen on port "28080": listen tcp :28080: bind: address already in use
./webserver.2020-12-18_165440.log:2020-12-18 16:43:06 [CRITICAL]failed to listen listen tcp :8899: bind: address already in use
./webserver.2020-12-18_165440.log:2020-12-18 16:43:06 [CRITICAL]failed to listen listen tcp :8899: bind: address already in use

- the solution was to look for a process that is using port 28080 or 8899 on the host that didn't report the agents back
Please note: This HAS TO be done as root. Otherwise as gpadmin you don't have a privileges to get a processes started as root or other user and you won't find out that they occupy the port. Always check netstat as root user.

netstat -nalp | grep -e '28080\|8899'

- once we found the process, it was killed and the gpcc agent was able to start as normal
- the permanent solution is to use a different port for either a process that was found and killed or change the gpcc configuration file and change the 28080 port to something unused (more in https://gpcc.docs.pivotal.io/4110/topics/configref.html)