Vertica database node down with error VAssert(!ctx.hasSnapshot) failed

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

tried to restart vertica, but one node fails to come up.

Admintools shows this message:

Nodes UP: v_drdata_node0002, v_drdata_node0001

Nodes DOWN: v_drdata_node0003 (may be still initializing).

Found these errors in startup.logs on hosts:

host ['xxx.xxx.xxx.xxx'] report: @v_drdata_node0003: VX001/5422: VAssert(!ctx.hasSnapshot) failed

        LOCATION:  Session::ScopedCatalogContext::~ScopedCatalogContext(), /scratch_a/release/svrtar14870/vbuild/vertica/Session/SessionContext.h:1515

Do you want to continue waiting? (yes/no) [yes]

Environment

Release : 21.2

Component : CA MEDIATION MANAGER

Cause

vertica.log shows the complete message:

2021-07-19 10:24:48.996 CatalogCheckpointer:7f5278ff9700 <ERROR> @v_drdata_node0003: {threadShimNoSession} 53100/2927: Could not write to [/catalog/drdata/v_drdata_node0003_catalog/Catalog/Checkpoints/c55267537_38494981/chkpt_local_2.cat.gz]: Volume [/catalog/drdata/v_drdata_node0003_catalog/Catalog] has insufficient space.

            LOCATION:  reserveSpace, /scratch_a/release/svrtar14870/vbuild/vertica/Workload/ResourceManager.cpp:4816

2021-07-19 10:24:48.997 CatalogCheckpointer:7f5278ff9700 <PANIC> @v_drdata_node0003: VX001/5422: VAssert(!ctx.hasSnapshot) failed

…

df -h showed that /catalog was 100% full

Resolution

Free up space on /catalog and restart node 3 using admintools

Restart vertica on host

Additional Information

du -h can help to find what is using up your free space.