Data Repository Server Crashing Over-Night

book

Article ID: 210935

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration

Issue/Introduction

Data Repository Server stops in the early hours and therefore Data Aggregator stops as well.  Next morning the Vertica server (DR) is restarted followed by DA server.  From the verticad.log we can see the following messages:

2021-03-16 03:15:49.364 Init Session:7f4dc37fe700-a000000195ffed <FATAL> @v_datarepository_node0001: 57V03/4748: Shutdown in progress. No longer accepting connections
 LOCATION:  runSession, /scratch_a/release/svrtar14870/vbuild/vertica/Session/ClientSession.cpp:1178
2021-03-16 03:15:49.364 Init Session:7f4f7f5e0700-a000000195fff0 <WARNING> @v_datarepository_node0001: 01000/4539: Received no response from v_datarepository_node0003 in roll back transaction
2021-03-16 03:15:49.364 Init Session:7f4e76ffd700-a000000195ffeb <WARNING> @v_datarepository_node0001: 01000/4539: Received no response from v_datarepository_node0003 in roll back transaction
2021-03-16 03:15:49.364 Init Session:7f4ebeffd700-a000000195fff3 [Txn] <INFO> Begin Txn: a000000195fff3 'SELECT /*+label(JDBC_CONNECTION_VALIDATION)*/ 1 FROM DUAL'
2021-03-16 03:15:49.364 Init Session:7f4f7f5e0700-a000000195fff0 <FATAL> @v_datarepository_node0001: 57V03/4748: Shutdown in progress. No longer accepting connections
 LOCATION:  runSession, /scratch_a/release/svrtar14870/vbuild/vertica/Session/ClientSession.cpp:1178

 

Cause

The cause was the server administrator was performing snapshots of the Vertica servers but not the Vertica partitions.  Snapshot was probably freezing the partitions. 

Environment

Release : 20.2

Component : APM Agents

Resolution

Do not perform snapshots of any Performance Management servers.  The preferred method for Vertica backups is to use the Vertica backup script.

Additional Information

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-netops/20-2/back-up-the-data-repository.html