AIOps - dxi-grafana pod in CrashLoopBackoff

Products

• DX Operational Intelligence
• DX Application Performance Management

Issue/Introduction

After on-prem installation of 24.1, dxi-grafana pod in CrashLoopBackoff 

Environment

• AIOps 24.x
• DX Platform 24.x
• DX Application Performance Management 24.x

 

Cause

Due to connectivity issues on the environment, the database scripts for the grafana pod failed,
As a result the data setup/migration of grafana tables did not complete successfully hence the grafana POD is in CrashLoopBackoff 

Verification:

Open postgres pod terminal and run the below queries

psql grafana_db aopuser;
select count(*)  from dashboard_version;
select count(*)  from migration_log;
select *  from migration_log where migration_id like 'save existing dashboard %';

IMPORTANT : the expected # of rows for dashoard_version is 239 and for migration_log is 305

 

Resolution

1) Scale down to 0 grafana pods: dxi-grafana and dxi-grafana-reporting-engine

If openshift, go to the console > deployments

2) Delete data in dashboard_versions:

delete from dashboard_version;

3) Scale up to 1 grafana pods: dxi-grafana and dxi-grafana-reporting-engine

4) Wait for 2 minutes, check there is no exceptions in dx-grafana log

5) Open postgres pod terminal again and run the below queries for verification:

psql grafana_db aopuser;
select count(*)  from dashboard_version;
select count(*)  from migration_log;

6) Next step is to fix the registration of the tenant because of the grafana pod issue.

When trying to access dx-dashboard you will see "Server Error" "organization not found"

a) Access http(s)://dxi-dashboards.<your-domain>/login

b) Go to Server > Orgs

c) Click New Org, add the CohortID of your tenant

You can obtain this ID from the connector parameter page 

Click "Create" button

7) Wait for a couple of minutes

8) Open DX OI in a new browser window, open DX Dashboards and it should open without any error.

 

 

 

Additional Information

AIOPs - Troubleshooting, Common Issues and Best Practices