DX Dashboards - Scheduled reports are not sent

book

Article ID: 242037

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

I have scheduled a report on DX Dashboard and no email was sent however I can preview the report and see it as PDF.

 

Environment

DX Platform 2x

 

Resolution

A) CHECKLIST

 
1. Reporter engine generates reports. This is run every 15 minutes.
 

2. Since you are scheduling at 15 th minute in the hour, see the following statements in dxi-grafana-deployment log
 
t=2022-05-19T06:15:01+0000 lvl=info msg="Fetched reports from database" logger=reporting.scheduleReader
t=2022-05-19T06:15:01+0000 lvl=info msg="Execution Queue getting updated for " logger=reporting.scheduler scheduleCount=0
 
Actions to take:
The schedule should be picked up so count should be greater than 0
 If you dont find these statements please check if the DXI_GRAFANA_PASSWORD environment variable is available in the grafana pod. And it has a base64 encoded value of grafana admin password
 

3. Once you confirm the schedule is picked up, see subsequent lines to see your report name exists in the list. See all the four statements exist for your schedule.
 
"Scheduler: Putting job on to exec queue" logger=reporting.scheduler name="report1" id=48 
t=2022-05-19T08:00:01+0000 lvl=info msg="Job picked for processing " logger=reporting.engine scheduleName=" report1"
t=2022-05-19T08:00:01+0000 lvl=info msg="Job being processed " logger=reporting.engine scheduleName="report1" attemptID=1
t=2022-05-19T08:00:01+0000 lvl=info msg="Calling report generator for first attempt" logger=reporting.engine Schedule Name="report1"
 
Actions to take:
If the schedule is not getting picked, please check the time and timezone in grafana and reporter pod are are in sync
 
 
4. Scroll to a place where it has the statements
 
t=2022-05-19T08:00:18+0000 lvl=info msg="outside lock for report1" logger=reporting.reportHandler
t=2022-05-19T08:00:18+0000 lvl=info msg="Handling report using result handler" logger=reporting.engine scheduleName="report1"
t=2022-05-19T08:00:18+0000 lvl=info msg="Result handler" logger=reporting.resultHandler scheduleName="report1"
t=2022-05-19T08:00:18+0000 lvl=info msg="Sending report notification to" logger=reporting.notifier [email protected]
t=2022-05-19T08:00:18+0000 lvl=info msg="Building email message for schedule" logger=notifications schedule="DX Dashboards, Report for report1"
t=2022-05-19T08:00:22+0000 lvl=info msg="Setting the job status to finished " logger=reporting.engine scheduleName="report1"
In this part it will either be success or a proper message would be shown why the report is failed.
 
Actions to take:
A proper message would be shown why the report is failed if not a success
 

B) COMMON ISSUES

PROBLEM #1: wrong smtp details

 
Recommendation:
 
kubectl edit deployment <dx-grafana-deployment> -n<namespace>
 
Update environment variables:
GF_SMTP_HOST smtp://<SMTP HOST/IP>:25
 
 
 

PROBLEM #2: Problem connecting to reporter pod.

 
below error message is reported in dxi-grafana-deployment log:

"t=2022-05-19T08:15:01+0000 lvl=eror msg="Remote reporting service responded with error." logger=reporting.reportHandler error="Get \"http://dxi-grafana-reporter.acme.local/api/v5/report/K9Ud24unk?gridlayout=false&from=now-1h&to=now\": dial tcp: lookup dxi-grafana-reporter.acme.local: no such host"

Recommendation:

- Go inside grafana-deployment pod

- Verify you can reach grafana reporter: curl http://dxi-grafana-reporter <fqdn>/health/live

- If dxi-grafana-reporter is not available edit grafana-deployment

kubectl edit deployment <grafana-deployment> -n<namespace>

- Add below section and lines:

hostAliases:
        - hostnames:
            - <dxi-grafana-reporter router>
          ip: <dxi-grafana-reporter IP>

Example:

hostAliases:
        - hostnames:
          - dxi-grafana-reporter.headq-acme.local
          ip: 10.99.61.79

 
 

PROBLEM #3 : certificate issues

 
Recommendation:
 
kubectl edit deployment <dx-grafana-deployment> -n<namespace>
 
Update environment variables:
GF_SMTP_SKIP_VERIFY true
 
 

PROBLEM# 4: Wrong password , unauthorized

 
Recommendation:
 
Correct DXI_GRAFANA_PASSWORD variable, this should be the superadmin password base 64 encoded
 
 

C) What diagnostic files should I gather for CA Support?

 
If you need further assistance contact Broadcom Support and provide below information:
 
- copy of dxi-grafana-deployment log : kubectl  logs <dxi-grafana-deployment-pod> -n<dix-namespace>
- kubectl describe po <dxi-grafana-deployment> -n<dxinamespace>
- kubectl get pods -n<dxinamespace>
- kubectl get events -n<dxinamespace>
 

Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html