vCenter Server 8.0 vPostgres service fails to start with PANIC: could not locate a valid checkpoint record

Products

VMware vCenter Server 8.0

Issue/Introduction

After patching or rebooting vCenter Server 8.0, the vmware-vpostgres service fails to start.
Running service-control --status shows vmware-vpostgres in a stopped state.
The /var/log/vmware/vpostgres/postgresql.log contains the following error: PANIC: could not locate a valid checkpoint record
The /storage/db partition appears to be mounted on a snapshot volume (e.g., /dev/mapper/db_vg-db_vg_db_snap) instead of the original logical volume (/dev/mapper/db_vg-db).

Environment

vCenter Server 8.x

Cause

This issue occurs because the vCenter appliance mistakenly mounts a leftover LVM snapshot volume for /storage/db instead of the production logical volume. Because the snapshot's pg_control file contains a stale checkpoint that does not match the live Write-Ahead Logs (WAL) on the primary volume, PostgreSQL fails to start to prevent data corruption.

Resolution

To resolve this issue, you must unmount the snapshot and remount the correct production volume.

Prerequisites

Ensure you have a fresh backup or offline snapshot of the vCenter Server Appliance before proceeding.

Validation Procedure

Identify the current mount point for /storage/db:
```
df -h /storage/db
```
Confirm if the filesystem points to a snapshot volume like db_vg_db_snap.
Stop all vCenter services:
```
service-control --stop --all
```
Verify production data integrity on the original volume (read-only check)
3a. Create a mount point Create a temporary directory to use for mounting the production volume without impacting the current system state:
```
mkdir -p /mnt/dbcheck
```
3b. Mount the production volume (Read-Only) Mount the original production volume (/dev/mapper/db_vg-db) to the temporary directory. The -o ro,noload flags ensure the volume is mounted as read-only and prevents the database from attempting to load or perform recovery actions, keeping the data untouched:
```
mount -o ro,noload /dev/mapper/db_vg-db /mnt/dbcheck
```
3c. Verify file system metadata Check the metadata for the pg_control file. This confirms the file exists and allows you to view the "Modify" timestamp to see if it reflects a recent checkpoint:
```
stat /mnt/dbcheck/vpostgres/global/pg_control
```
3d. Inspect PostgreSQL cluster data Use the pg_controldata utility to inspect the internal state of the database cluster. Grepping for the "cluster state" and "time of latest checkpoint" allows you to verify that the data on this volume is current and consistent:
```
/opt/vmware/vpostgres/current/bin/pg_controldata /mnt/dbcheck/vpostgres | grep -iE "cluster state|time of latest checkpoint"
```
3e. Clean up temporary mount After validation, unmount the temporary directory to release the production volume:
```
umount /mnt/dbcheck/vpostgres
```

Resolution Procedure

Unmount the temporary check partition.
```
umount /mnt/dbcheck
```
Identify and terminate any processes still accessing the /storage/db mount point.
```
fuser -vm /storage/db
```
Unmount the stale LVM snapshot volume.
```
umount /storage/db
```
Mount the correct production volume: This command will make changes to your system. Review it carefully before running.
```
mount -o nosuid,nodev,noatime,nodiratime /dev/mapper/db_vg-db /storage/db
```
Verify the mount and file status: Review this command before running it.
```
df -h /storage/dbstat /storage/db/vpostgres/global/pg_control
```
The 'Modify' date should now reflect the most recent checkpoint found in step 3.
Start vCenter services: This command will make changes to your system. Review it carefully before running.
```
service-control --start --all
```
Clean up dangling snapshots: Once the vCenter is stable, run the LVM cleanup script that is attached to KB370818 to prevent recurrence.