Encountering FATAL: DTM initialization: failure during startup recovery Error on Query Attempt

Products

• VMware Tanzu Greenplum

Issue/Introduction

When trying to perform database operations (ie. open connection or psql session), the error below is encountered:

FATAL: DTM initialization: failure during startup recovery, retry failed, check segment status (cdbtm.c:1568)

The errors below may be seen in the master log:

"segment is in recovery mode (segXXX XX.XXX.XX.XX)"

"FTS: segment (dbid=XXX, content=XXX) has not reached new state yet, expected state: ('X','X'), reported state: ('X','X')

"DTM initialization, caught exception: looking for failed segments."

Looking at the primary segment logs for the segment mentioned in the above errors:

"mirror failure, no connection was established from client to mirror, primary address(port '<PRIMARY-HOST>(<PORT>)' mirror address(port) '<MIRROR-HOST>(<PORT>)' timeout reached '600' failover requested"

Environment

Product Version: 5.28

Resolution

The errors above indicate a lack of connection between primary and mirror hosts.

Check the health of the connections between primary->mirror and mirror->primary through ssh and ping — connection may be only working one-way.

Also check connection between master to each host as well as hosts to master.