GPCC metrics-collection agents fail to start after upgrade to 6.17.1 (Network Unreachable)

Products

• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

Symptoms

After upgrading Greenplum Command Center (GPCC) from 6.11.x to 6.17.1, the metrics-collection agents (ccagent) on segment hosts fail to start. The following error is observed in the segment agent logs:

[ERROR] ControlCommand: agent cannot start due to no RPC connection failure rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp <COORDINATOR_MGMT_IP>:8899: connect: network is unreachable"

Key Observations

• The agent attempts to connect to the coordinator's management/external IP instead of the internal interconnect IP.
• In previous versions (e.g., 6.11.1), the agent correctly used the hostname or internal IP.
• This typically occurs in multi-homed environments where the coordinator's default route is on a management network that segment hosts cannot route to.

Cause

In GPCC 6.17.1, an architectural change was introduced in how the application handles network advertisement. The backend may auto-detect the coordinator's RPC server address from the default-route interface rather than honoring the master_host or gpcc_host settings in app.conf or the Greenplum catalog.

Resolution

You can override the auto-discovery behavior and pin gRPC communications to the internal cluster network using the grpc_ip_allowlist parameter.

Step-by-Step Implementation

1. Log in to the coordinator node (mdw) as the gpadmin user.
2. Open the GPCC configuration file:

   vi $GPCC_HOME/conf/app.conf

3. Add the grpc_ip_allowlist parameter at the bottom of the file. Include the local loopback address and your internal cluster interconnect subnet (CIDR notation):
   grpc_ip_allowlist = mdw, smdw, sdw1, sdw2, sdw3, sdw4
4. Restart Greenplum Command Center:
   gpcc stop

   gpcc start

Verification

Check the agent log on any segment host to confirm the successful connection:

tail -n 20 $GPCC_HOME/logs/agent.log

Expected output:

[INFO] connect to rpc server mdw:8899[INFO] ControlCommand: connection established mdw:8899