GPDB 7 TDE Initialization Fails - Mirror Segments Down with "invalid magic number"

Products

VMware Tanzu Data Suite

Issue/Introduction

Issue Description

When attempting to initialize a Greenplum Database (GPDB) 7 cluster with Transparent Data Encryption (TDE) enabled, the gpinitsystem process fails. The primary segments start successfully, but the mirror segments crash, fail to synchronize, and cannot be recovered.

This issue typically occurs in testing environments where a local Key Management Service Key (KMSK) file is used across multiple hosts rather than a centralized KMS server.

Error Messages

1. gpinitsystem Log: The initialization script will eventually fail and report that primary/mirror pairs are out of sync:

20260811:19:29:14:1176354 gpinitsystem:<HOSTNAME>:gpadmin-[WARN]:-Failed to start Greenplum instance; please review gpinitsystem log to determine failure.
20260811:19:29:14:1176354 gpinitsystem:<HOSTNAME>:gpadmin-[FATAL]:-Some primary/mirror segment pairs were found to be not in sync Script Exiting! 

2. Mirror Segment Log (pg_log): If you inspect the logs of the failed mirror segment, you will see it repeatedly failing to decrypt the Write-Ahead Log (WAL) files streamed from the primary, resulting in an invalid magic number error from xlog.c:

2026-08-11 19:29:03.086878 WIB,,,p639658,th894661376,,,,0,,,seg10,,,,,"FATAL","57P01","terminating walreceiver process due to administrator command",,,,,,,0
,,"walreceiver.c",167,
2026-08-11 19:29:03.187212 WIB,,,p639657,th894661376,,,,0,,,seg10,,,,,"LOG","00000","invalid magic number 5923 in log segment 000000010000000000000002, offs
et 0",,,,,,,0,,"xlog.c",13364, 

Environment

Product: Greenplum Database 7.x (e.g., 7.8.3)
Feature: Transparent Data Encryption (TDE)
Configuration: TDE is enabled using a local Master Key file distributed across segment hosts.

Cause

Root Cause

The root cause is a Master Key (KMSK) mismatch across the cluster hosts.

During initialization, the Primary segment generates an internal Segment Data Encryption Key (SDEK) and wraps it using the local Master Key on Host A. The Primary then uses this key to encrypt its WAL files. When the Mirror segment on Host B attempts to start, it receives the encrypted WAL files and tries to load the SDEK using the Master Key located on Host B. Because the master key files on Host A and Host B have different contents, the Mirror silently fails to unwrap the correct internal key. Consequently, it attempts to decrypt the WAL files with the wrong key, resulting in corrupted data and the invalid magic number error.

Resolution

When using a local file for the KMSK, the key file must be identical on every host in the cluster.

Verify the mismatch: Run an MD5 checksum on the master key file across all hosts.
```
gpssh -f hostfile "md5sum /path/to/the/keyfile"
```
If the hashes do not match, proceed to step 2.

Synchronize the key: Copy the correct Master Key from the coordinator (master) node to all segment hosts and ensure strict permissions.

gpscp -f hostfile /path/to/the/keyfile =:/path/to/the/keyfile
gpssh -f hostfile "chown gpadmin:gpadmin /path/to/the/keyfile && chmod 400 /path/to/the/keyfile"

Re-initialize: Clean up the failed cluster installation using gpdeletesystem (or manually clear the data directories) and re-run gpinitsystem.
Verify TDE status: Once the cluster is running, confirm TDE is healthy across all segments:
```
gpctl tde status
```

Additional Information

Technical Background: How TDE Works in GPDB

TDE utilizes a two-tier key architecture:

SDEK (Segment Data Encryption Key): Generated automatically in plaintext by the database during initialization. It resides in memory to perform the actual encryption/decryption of data and WAL files.
KMSK (Key Management Service Key): The user-managed "Master Key" used to protect the SDEK.

Configuration Workflow (`gpinitsystem_config`)

TDE_ALGORITHM: Enables TDE (e.g., aes-128-ctr).
TDE_KEY_WRAP_COMMAND: Executed during cluster initialization. The database passes the plaintext SDEK to this command, which uses the KMSK to encrypt (wrap) it into a binary format. The result is stored on disk in <DATA_FOLDER>/data_encryption.key.
TDE_KEY_UNWRAP_COMMAND: Executed during segment startup. The database reads <DATA_FOLDER>/data_encryption.key, passes it to this command to decrypt (unwrap) it using the KMSK, and stores the plaintext SDEK back into memory.

⚠️ Critical Security Warnings

Data Loss Risk: If the Master Key (KMSK) is lost, deleted, or accidentally overwritten (e.g., during a botched key rotation), all encrypted data will be permanently lost. There is no backdoor to recover the data without the original key.
Production Best Practices: While local OpenSSL keys (as shown below) are acceptable for testing, production environments should always utilize a centralized, enterprise Key Management Service (KMS) to manage the Master Key.

Example of a local key setup for testing purposes only:

# Generate Key
openssl rand 32 > /home/gpadmin/tde_master.key
chmod 400 /home/gpadmin/tde_master.key

# gpinitsystem_config
TDE_ALGORITHM=aes-128-ctr
TDE_KEY_UNWRAP_COMMAND="openssl enc -d -aes-128-ctr -pbkdf2 -pass file:/home/gpadmin/tde_master.key"
TDE_KEY_WRAP_COMMAND="openssl enc -aes-128-ctr -salt -pbkdf2 -pass file:/home/gpadmin/tde_master.key"

References:

Tanzu Greenplum 7 Security Guide - Transparent Data Encryption