DX AIOPs Troubleshooting, Common Issues and Best Practices
Article ID: 190815
Updated On:
Products
DX Operational Intelligence
DX Application Performance Management
CA App Experience Analytics
Issue/Introduction
The following is a high-list of techniques and suggestions to employ when troubleshooting performance, display and configuration issues
A) Openshift / Kubernetes
B) DX Platform Installer
C) Data Platform (Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)
D) DX Dashboards
E) DX Operational Intelligence
F) DX Application Performance Management
Environment
DX Operational Intelligence 1.3.x, 20.x
DX Application Performance Management 11.x, 20.x
DX AXA 20.x
Resolution
Table of Contents
A) Openshift / Kubernetes
- DX AIOps - Unable to open Openshift console
- DX AIOps - Unable to login to OI console, all Openshift nodes appear in a NotReady status
- DX AIOps - many pods in Evicted status - how can I delete these unwanted pods?
B) DX Platform Installer
Installation
- DX Platform - Example how to install Openshift 3.11 + DX 20.2.1 Full Install (template 4 servers x 64GB Mem)
- DX Platform - Example how to Install Openshift 3.11 + DX 20.2.1 OI+APM Only (template 6 servers x 64GB Mem) + 3 ElasticNodes
- DX Platform - Example how of Install Kubernetes + DX 20.2.1 Full Install (template 5 servers x 64GB Mem)
- DX Platform Installer fails with [elasticsearch-node-validate] [ERROR] No nodes were detected in the cluster.
NFS
C) Data Platform (Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)
Login issues
- DX Platform - Unable to login due to PODs connectivity issues
- DX Platform - Unable to login due to USER PASSWORD IS LOCKED
- DX Platform - Unable to login due to password expiration
- DX Platform - Unable to login, error message "502 Bad Gateway"
- DX Installer - Unable to login after DX installation due to dxi-platform-init-pod errors
- DX Platform - Unable to login - How to reset masteradmin password if forgotten
- DX Platform - Unable to login to newly created Tenants - login process enter in a loop
- Login issue as masteradmin on DX Platform 20.2.1 after install
- DX Platform - Unable to login due to postgres database corruption
- DX Platform - Unable to login to DX Platform - jarvis-elasticsearch pods evicted
Jarvis (kafka, zookeeper, elasticSearch)
- DX Platform - ElasticSearch Queries
- DX Platform - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
- DX Platform - NFS or Elastic Nodes disk full - How to reduce Elastic data retention
Cluster Management
HOTFIXES
Maintenance
D) DX Dashboards
E) DX Operational Intelligence (OI)
Alarms Analytics
- DX OI - Unable to use the Filter option from "Alarm Analytics", it shows "Error Retrieving" message
- DX OI - No New alarms or Alarms are not getting updated
- DX OI - Is it possible to customize the Alarm monitoring details from OI console or a config file?
- DX OI - (Enhancement) Be able to automate the way of assigning the events based on alarm type/Description/Sub description
- Make a saved filter public (limitation)
Anomaly Alarms
Performance Analytics
- DX OI - Metrics are not visible in Performance analytics and Alarms are not reported
- DX OI - How to delete unwanted metrics from Performance Analytics UI
- DX OI - New Tenants reporting ERROR "400 /oi/v3/queryTas/analytics" and "500 /oi/v2/api/Inventory/fields"
- DX OI - Performance Analytics page throws message : 500 /oi/v2/api/inventory/fields
- DX OI - Predictive Insights stop working, the page is empty
- DX OI - Unable to see CAPM Inventory data from Performance Analytics
- OI green highways not displaying correctly for custom metrics
Predictive Insight
- DX OI - How to export Capacity Analytics and Predictive Insight data?
- Capacity forecast analytics show CPU consumption 6000% for next 6 month
- Capacity Analytics error messages
Service Analytics
Integration with APM 10.7
- DX OI integration with APM 10.7 - Troubleshooting, Common Issues and Best Practices
OI 20.x :
- DX OI - Alarms explosion caused by the APM integration - How to delete All or individual APM alarms?
- DX OI - "APM Alarms do not support the action" message, unable to clear or hide any APM Alarm in OI UI
Integration with NetOps
- DX OI integration with DX NetOps PM, CA ADA, and CA NFA - OI Connector Troubleshooting, Common Issues and Best Practices
- DX OI - OIConnector not connecting when CAPC user and password changes
Integration with UIM
- DX OI integration with UIM - Troubleshooting, Common Issues and Best Practices
OI 20.x : https://knowledge.broadcom.com/external/article/207711
OI 1.3.x : https://knowledge.broadcom.com/external/article/144181
- DX OI integration with UIM - Disaster Recovery for OI Connector and APM Bridge
- DX OI integration with UIM - apm_bridge is not sending data : Could not determine UIM or TAS devices to forward for profile 0 with origin
- DX OI integration with UIM - duplicated alarms and some alarms are not cleared property in OI Console
- DX OI - UIM integration, user tags missing as part of the UIM Alarms
- UIM alarms are not reflecting the colored status on the UIM device in the Service Topology view
- The edit service or Update service(Synch for group/Inventory) are not supported.
Integration with Spectrum
- DX OI - Spectrum Data Publisher not sending inventory, error code: 503
- Spectrum topology not properly synced
- mapping Spectrum alarms to services
ServiceNow
- DX AIOps - ServiceNow Integration Troubleshooting
- DX AIOps - Nim Troubleshooting
- DX AIOps- How to manually recreate a tenant configuration in NIM UI?
- DX Platform - How to create a NIM route to access to the NIM UI?
- DX OI - Unable to create a ServiceNow incident "1 alarm action(s) failed to perform"
- DX OI - ServiceNow Automatic incident creation/closure is not working
- DX OI - Alarms do not display the generated ServiceNow incidents
- DX OI - Many ServiceNow incidents and Service Alarms are being created
- DX OI - ServiceNow incidents are getting reopened many times.
- DX OI - ServiceNow incidents are created but OI console still shows the “Open ticket” option rather than the incident number
- DX OI - How to assign an event to a particular group in ServiceNow based on CI's like Process, Service or URL.
- DX OI - Is it possible to create a new custom Field in ServiceNow incident that includes the UIM process name?
REST API
Self Service Dashboards
F) DX Application Performance Management (APM)
Business Transactions
Upgrade
ACC Command Center
- DX APM - ng-acc-configserver-deployment pod in CrashLoopBackOff due to FATAL: password authentication failed for user "acc"
- DX APM - ng-acc-configserver pods in CrashLoopBackOff due to ACC postgres DB corruption
Tenant data migration
Additional Information
Installation
DX OI
DX APM
Feedback
Yes
No
Powered by
