Greenplum: How to safely replace segment host/server hardware (RAM, CPU, NIC, Raid Controller, etc)
Article ID: 453538
Updated On:
Products
Issue/Introduction
This knowledge base article details the procedure for replacing hardware on a Greenplum segment host without causing a cluster outage. Because Greenplum utilizes primary/mirror segment pairs for High Availability, a controlled, graceful shutdown of a host allows the cluster to automatically failover and continue
serving queries.
Critical Warning: Double-Fault Risk
If you forcefully shut down a segment host while mirrors on other hosts are already degraded or resynchronizing, you risk a double-fault (losing both the primary and its corresponding mirror). This will result in an immediate cluster outage. Always verify cluster health before proceeding.
Environment
Greenplum 5.x, 6.x, 7.x
Cause
There are cases when defective server hardware needs to be replaced/repaired without impacting database operations. Resolution
1. Pre-Maintenance: Verify Cluster Health
Before initiating any host shutdown, you must confirm that all segments are functioning and fully synchronized.
Execute the following commands as the gpadmin user from the master (coordinator) node:
gpstate -e
gpstate -m
gpstate -e should return no errors.
gpstate -m should list all segments as Synchronized. Do not proceed if any segments are in Change Tracking or Resynchronizing modes.
2. Gracefully Stop a Single Target Host.
To safely isolate the host for maintenance, initiate a graceful shutdown of the segment instances on the target machine. The utility shuts down the Greenplum Database segment instances on the specified host to allow maintenance on the host. Each primary segment instance on the host is shut down and the associated mirror segment instance is promoted to a primary segment if the mirror segment is on another host. Mirror segment instances on the host are shut down.
gpstop --host <target_hostname>
3. Check the status to confirm segments on the target host are marked down.
Once the command completes, run gpstate -s. The primary and mirror. segments on the target host should now be marked as down (d), and the cluster will be operating in change tracking mode using the promoted mirrors.
gpstop documentation: https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-greenplum/5/greenplum-database/utility_guide-admin_utilities-gpstop.html
4. Perform hardware replacement.
If you are replacing components but retaining the original hard drives and OS installation, simply complete the physical swap, power the machine back on, and ensure the host is reachable via SSH from the master node.
If you are swapping the entire physical server or wiping the disks, you must ensure the environment matches the previous state:
Network: The new server must use the exact same Hostname and IP address(es).
Storage: Data directories (e.g., /data/primary, /data/mirror) must be mounted with the samepaths and owned by the gpadmin user.
Software: Install the Greenplum binaries to the identical path as the previous host.
Authentication: You must recreate the gpadmin user and restore SSH trust by running gpssh-exkeys -h <target_hostname> from the master node.
5. Recover the segments.
Run an incremental recovery. This will only copy the data that changed while the host was offline.
gprecoverseg -a
6. Monitoring the Recovery:
Run gpstate -m to view the synchronization progress. The cluster remains fully operational for user queries during this background synchronization.
7. Rebalance the Cluster
The cluster is currently running with the target host serving as a mirror for segments it usually owns as a primary. You must rebalance the roles back to their preferred state. Wait until synchronization is complete. Do not run the rebalance command until gpstate -m shows all segments are Synchronized.
Once synchronized, run the rebalance command:
gprecoverseg -a -r
8. Verify that all segments are up and operating in the preferred roles.
gpstate -s shows detailed status info.
gpstate -e
gpstate -s
Feedback
