Grafana pod in CrashLoopBackOff , error MultiXactId 1 has not been created yet -- apparent wraparound

Products

• DX Operational Intelligence

Issue/Introduction

Grafana is not coming up and the state plus the relevant output from the logs show below.

dxi-grafana-6fb9866c59-pxwqb                             0/1     CrashLoopBackOff   15 (4m25s ago)   57m

dxi-grafana-reporting-engine-bcb9b4ddc-4v9dg   0/1     CrashLoopBackOff   15 (4m31s ago)   57m

 

Relevant log entries for both pods.

service init failed: could not determine if admin user exists: pq: MultiXactId 1 has not been created yet -- apparent wraparound

Postgres logs keep getting the error when running the query SELECT COUNT(id) AS Count FROM "user"

 

Environment

DX Platform 23.x, 24.x

Cause

Table public.user in kibana_db was corrupt.

 

Resolution

Any attempt to run a query showed the same error.

This table, public.user, in grafana_db is automatically created/populated when grafana pod starts so it can be truncated.

Steps to truncate:

    - Open a bash to the grafana schema pod

    - Run script run_psql_grafana_db.bash (it will connect to PostgreSQL)

    - Run this commands in the psql prompt

        \c grafana_db <Enter>

       truncate public.user; <Enter>

      vacuum full public.user, <Enter>

Restart grafana and grafana schema pods 

Additional Information

Vaccum analyze and vacuum full did not make any difference.