Vertica DB Node Down and Unable to Start after PANIC

Products

• Network Observability
• CA Performance Management
• Virtual Network Assurance

Issue/Introduction

One of our vertica nodes went down, and we are unable to start it even using the force (--force, -F) options

 

errors seen in vertica.log

……..

2026-07-08 09:39:59.329 SAL Record Check Thread:0x7ef6867fc700 [Init] <INFO> Startup [Check Storage] Confirming storage matches catalog (files) - 59274 / 98712

2026-07-08 09:40:00.151 SAL Record Check Thread:0x7ef6acff9700 [SAL] <WARNING> Error getting size of bundle file of [02c7e025532f25adbc5722da0000340589b1d]: StorageBundleReader: Error reading metadata of file [/<path>/drdata/v_drdata_node0001_data/549/02c7e0255bc5722d2b63340589b1d_0.gt]: Bundle metadata is corrupted 

2026-07-08 09:40:00.151 SAL Record Check Thread:0x7ef6acff9700 <WARNING> @v_drdata_node0001: 01000/3938: MiniRos <value> does not have proper SAL files

…………….

2026-07-08 09:40:06.718 Main:0x7efd55dcff0000000000cc1 <PANIC> @v_drdata_node0001: VX001/2973: Data consistency problems found; startup aborted

HINT:  Check that all file systems are properly mounted.  Also, the --force option can be used to delete corrupted data and recover from the cluster

LOCATION:  mainEntryPoint, /data/jenkins/workspace/RE-ReleaseBuilds/RE-Laserbeam/server/vertica/Basics/vertica.cpp:2160

2026-07-08 09:40:06.740 Main:0x7efd55dc9d80-fff0000000000cc1 [Main] <PANIC> Wrote backtrace to ErrorReport.txt

……………..

Environment

DX NetOps :: All Supported Release

VNA :: All Supported Release

 

Cause

Potential storage issues have caused corruption in the vertica DB on the affected node.

Resolution

• Reset the Ancient History Mark (AHM) to now, to force a scratch recovery of the node.

   admintools, connect to database.

   select make_ahm_now(true);

   \q to exit vertica 

 

• Start DB on node with force to initiate scratch recovery:

   /opt/vertica/bin/adminTools -t restart_node -F -s <IP> -d drdata