DX AIOPs - Troubleshooting, Common Issues and Best Practices
Article ID: 190815
Updated On:
Products
DX Operational Intelligence
DX Application Performance Management
CA App Experience Analytics
Issue/Introduction
The following is a high-list of techniques and suggestions to employ when troubleshooting performance, display and configuration issues
A) Openshift / Kubernetes
B) DX Platform Installer
C) Data Platform (Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)
D) DX Dashboards
E) DX Operational Intelligence
F) DX Application Performance Management
Environment
DX Operational Intelligence 1.3.x, 20.x
DX Application Performance Management 11.x, 20.x
DX AXA 20.x
Resolution
A) Openshift / Kubernetes
- DX AIOps - Unable to open Openshift console
- DX AIOps - Unable to login to OI console, all Openshift nodes appear in a NotReady status
- DX AIOps - many pods in Evicted status - how can I delete these unwanted pods?
B) DX Platform Installer
NFS
Sizing Calculator
Installation Templates
- DX Platform - Example how to install Openshift 3.11 + DX 20.2.1 Full Install (template 4 servers x 64GB Mem)
- DX Platform - Example how to Install Openshift 3.11 + DX 20.2.1 OI+APM Only (template 6 servers x 64GB Mem) + 3 ElasticNodes
- DX Platform - Example how of Install Kubernetes + DX 20.2.1 Full Install (template 5 servers x 64GB Mem)
Installation issues
- DX Platform Installation fails with [ ERROR ] Pod of Deployment .. has failed to start in the given timeframe
- DX Platform Installer fails with [elasticsearch-node-validate] [ERROR] No nodes were detected in the cluster.
- DX Installer - dxi-platform-init-pod logs contains errors
- DX Platform Installer doesn't update ingress with correct cert - Kubernetes Ingress Controller Fake Certificate
Post Installation
- DX Platform - How to enable High Availability for ElasticSearch?
- DX Platform - Unable to activate DX OI, APM or AXA using dxi-component-scale.sh
- DX Platform - dxi-es-admin.sh fails with error "Jarvis Elasticsearch Pod is not running"
- DX Platform - Is it possible to secure routes or enable SSL after installation?
- DX Platform - Is it possible to change the wildcard DNS entered during DX Platform installation
HOTFIXES
Maintenance
C) Data Platform (Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)
Login issues
- DX Platform - Unable to login due to PODs connectivity issues
- DX Platform - Unable to login due to USER PASSWORD IS LOCKED
- DX Platform - Unable to login due to password expiration
- DX Platform - How to change the default password expiration policy of 180 days?
- DX Platform - Unable to login, "502 Bad Gateway" error after adding a 2nd nic
- DX Installer - Unable to login after DX installation due to dxi-platform-init-pod errors
- DX Platform - Unable to login - How to reset masteradmin password if forgotten
- DX Platform - Unable to login to newly created Tenants - login process enter in a loop
- DX Platform 20.2.1 - Unable to login, login is successful but it immediately logout
- DX Platform - Unable to login due to postgres database corruption
- DX Platform - Unable to login to DX Platform - jarvis-elasticsearch pods evicted
Jarvis (kafka, zookeeper)
- DX Platform - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
- DX Platform - How to restart Jarvis services?
- DX Platform - kafka data consuming all disk space in Elastic nodes
- DX Platform - After creating a New tenant jarvis-verifier keeps crashing
ElasticSearch
Cluster Management
D) DX Dashboards
E) DX Operational Intelligence
Alarms Analytics
- DX OI - New alarms don't appear in OI Console and ServiceNow incident creation is not working
- DX OI - Unable to use the Filter option from "Alarm Analytics", it shows "Error Retrieving" message
- DX OI - No New alarms or Alarms are not getting updated
- DX OI - Is it possible to customize the Alarm monitoring details from OI console or a config file?
- DX OI - (Enhancement) Be able to automate the way of assigning the events based on alarm type/Description/Sub description
Situations
Performance Analytics
- DX OI - Metrics are not visible in Performance analytics and Alarms are not reported
- DX OI - How to delete unwanted metrics from Performance Analytics UI
- DX OI - New Tenants reporting ERROR "400 /oi/v3/queryTas/analytics" and "500 /oi/v2/api/Inventory/fields"
- DX OI - Performance Analytics page throws message : 500 /oi/v2/api/inventory/fields
- DX OI - Predictive Insights stop working, the page is empty
- DX OI - Unable to see CAPM Inventory data from Performance Analytics
- OI green highways not displaying correctly for custom metrics
DSP / DSP Integrator / Anomaly Detection
Predictive Insight / CPA / Capacity Analytics
- DX OI - cpa-projection pod reporting CrashLoopBackOff status
- DX OI - How to create a DX Dashboards based on Prediction (capacity or performance) alarms?
- DX OI - How to export Capacity Analytics and Predictive Insight data?
Service Analytics
Integrations
Integration with APM 10.7
-
- DX OI integration with APM 10.7 - Troubleshooting, Common Issues and Best Practices
Integration with NetOps
Integration with UIM
-
- DX OI integration with UIM - Troubleshooting, Common Issues and Best Practices
OI 20.x : https://knowledge.broadcom.com/external/article/207711
OI 1.3.x : https://knowledge.broadcom.com/external/article/144181
-
- DX OI - oi_connector dropping alarms - CIInformation not found for alarm
- DX OI integration with UIM - Disaster Recovery for OI Connector and APM Bridge
- DX OI integration with UIM - apm_bridge is not sending data : Could not determine UIM or TAS devices to forward for profile 0 with origin
- DX OI integration with UIM - duplicated alarms and some alarms are not cleared property in OI Console
- DX OI - UIM integration, user tags missing as part of the UIM Alarms
Integration with Spectrum
-
- DX OI - Many alarms are cleared in Spectrum however they remain opened in Alarm Analytics
- DX OI - Spectrum 10.4.3.x or 21.x integration validation process returns Jarvis and OIUrl FAILED
- DX OI - Spectrum Data Publisher not sending inventory, error code: 503
- Spectrum topology not properly synced
- mapping Spectrum alarms to services
Notifications (Emails, Webhooks, Ticketing)
- DX AIOps - ServiceNow Integration Troubleshooting
- DX AIOps - Nim Troubleshooting
- DX AIOps- How to manually recreate a tenant configuration in NIM UI?
- DX Platform - How to create a NIM route to access to the NIM UI?
- DX Platform - Unable to open Email Channel, instead we get an empty page
- DX OI - Be able to reopen existing resolved Incidents for APM alerts
- DX OI - Unable to create ServiceNow incidents manually
- DX OI - ServiceNow Automatic incident creation/closure is not working
- DX OI - Alarms do not display the generated ServiceNow incidents
- DX OI - Many ServiceNow incidents and Service Alarms are being created
- DX OI - ServiceNow incidents are getting reopened many times.
- DX OI - ServiceNow incidents are created but OI console still shows the “Open ticket” option rather than the incident number
- DX OI - How to assign an event to a particular group in ServiceNow based on CI's like Process, Service or URL.
- DX OI - Is it possible to create a new custom Field in ServiceNow incident that includes the UIM process name?
REST API
F) DX Application Performance Management (APM)
Traces Tab
Business Transactions
Upgrade
ACC Command Center
- DX APM - ng-acc-configserver-deployment pod in CrashLoopBackOff due to FATAL: password authentication failed for user "acc"
- DX APM - ng-acc-configserver pods in CrashLoopBackOff due to ACC postgres DB corruption
Tenant data migration
Additional Information
Installation
DX OI
DX APM
Feedback
Yes
No
Powered by
