AIOPs - Troubleshooting, Common Issues and Best Practices
Article ID: 190815
Updated On:
Products
DX Operational Intelligence
DX Application Performance Management
CA App Experience Analytics
Issue/Introduction
The following is a high-list of techniques and suggestions to employ when troubleshooting performance, display and configuration issues
A) Openshift / Kubernetes
B) DX Platform Installer
C) Data Platform (Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)
D) DX Dashboards
E) DX Operational Intelligence
F) DX RESTMon
G) DX Application Performance Management
H) DX App Experience Analytics
Environment
DX Platform Installer 2x
DX Operational Intelligence 2x
DX Application Performance Management 2x
DX AXA 2x
Resolution
Table of Contents
- Common Issues
Common Issues
A) Openshift / Kubernetes
- AIOps - Unable to open Openshift console
- AIOps - Unable to login to OI console, all Openshift nodes appear in a NotReady status
- AIOps - many pods in Evicted status - how can I delete these unwanted pods?
- AIOps - How to clear pods in Terminating status
B) DX Platform Installer
Performance Recommendations
- AIOps - Top 3 items that affect performance
- AIOps - How to verify throughput for NFS (IOPs and Speed)
- AIOPs 20.2.1 - NFS disk full due to Data Science Platform (DSP) feature
Sizing Calculator
DX Platform Installer
21.3.1:
- Example How to install Kubernetes + DX Platform 21.3.1 (OI+APM+AXA) + 3 ElasticNodes NFS + ClusterReader
- Example How to install Openshift 3.11 + DX Platform 21.3.1 (OI+APM) + 3 ElasticNodes NFS + ClusterAdministrator
- Example How to install Kubernetes + DX Platform 21.3.1 (OI+APM) + 3 ElasticNodes NFS + ClusterAdministrator
Installation issues
- AIOps 21.3.1 DX Platform Installer ignores the use of a different NFS mount for Postgres
- AIOps - Installation fails with "java.lang.IllegalArgumentException: Invalid default value"
- AIOps - dxi-platform-init-pod reports errors
- AIOps - Installation fails with [ ERROR ] Pod of Deployment .. has failed to start in the given timeframe
- AIOps - Installer fails with [elasticsearch-node-validate] [ERROR] No nodes were detected in the cluster.
- AIOps - Unable to activate DX OI, APM or AXA using dxi-component-scale.sh
- AIOps - dxi-es-admin.sh fails with error "Jarvis Elasticsearch Pod is not running"
- AIOps - Installation failed due to Postgres fails to startup : could not link file "pg_wal/xlogtemp.29"to "pg_wal/00000000001" : Input/output error
- AIOps - Is it possible to secure routes or enable SSL after installation?
- AIOps - Is it possible to change the wildcard DNS entered during DX Platform installation
Post Installation
- AIOPs - dxi_update_ingress_secret.sh incorrectly reports that "Tls is not enabled"
- AIOps - How to enable High Availability for ElasticSearch?
- AIOps - How to create jarvis apis and elasticsearch endpoints that are disabled by default?
- AIOps - apmservices-metricforward pod in Error/CrashLoopBackOff status and keeps restarting
Upgrade issues
Deploy New Images
Maintenance
C) DX Platform (Onboarding, Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)
Onboarding (DX Admin Emails)
- AIOps - "DX Admin"-email-link not working properly. "The system is currently down."
- AIOps - "DX Admin" emails sent during user creation uses "First name" instead of "User name"
Login issues
- DX AIOps 21.3.1 - Unable to connect to DOI UI - doi-adminui pod in crashloopback
- DX AIOps 21.3.1 - Unable to login - doireadserver pod in crashloopback
- AIOps - Unable to login, postgresql pod in CrashLoopBackOff
- AIOps 21.3.1 - Unable to login to newly created Tenants, the login process enter in a loop if the top domain name is ".local"
- AIOps - Unable to login after installation, dxi-platform-init-pod reports errors
- AIOps - Unable to login due to PODs connectivity issues
- AIOPs - Unable to login - How to reset "masteradmin" password if forgotten
- AIOps - Unable to login due to USER PASSWORD IS LOCKED
- AIOps - Unable to login due to password expiration
- AIOps - How to change the default password expiration policy of 180 days?
- AIOps - Unable to login, "502 Bad Gateway" error after adding a 2nd nic
- AIOps 20.2.1 - Unable to login, login is successful but it immediately logout
- AIOps - Unable to login due to postgres database corruption
Jarvis (kafka, zookeeper, kron)
- AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
- AIOps - The jarvis-kron pod keeps restarting and some elasticsearch indices are in yellow status.
- AIOps - How to create the Jarvis apis and elasticsearch endpoints in 21.x
- AIOps - How to restart Jarvis services?
- AIOps - kafka data consuming all disk space in Elastic nodes
- AIOps - After creating a New tenant jarvis-verifier keeps crashing
- AIOps - kafka crashloop with cluster.id mismatch in meta.properties
ElasticSearch
Postgres
- AIOps - How to list databases in postgres?
- AIOps - Unable to login due to postgres database corruption
- AIOps - Unable to login, postgresql pod in CrashLoopBackOff
NAS
REST API
Policies
AUTOMIC
- AIOps - Example How to enable Automic integration
- AIOps 21.3.1 - AUTOMIC Policies are not working, remediation workflows are not executed
Notifications (Emails, Ticketing, Webhooks)
Emails:
Ticketing / ServiceNow:
- DX AIOps - ServiceNow Integration Troubleshooting
- DX AIOps - Nim Troubleshooting
- DX AIOps- How to manually recreate a tenant configuration in NIM UI?
- DX Platform - How to create a NIM route to access to the NIM UI?
- AIOps - Changes in the DefaultTicketingManagementTemplate message template are not reflected in ServiceNow
- DX OI - ServiceNow Automatic incident creation NOT working however manual creation is working
- DX OI - Unable to create ServiceNow incidents manually
- DX OI - ServiceNow Automatic incident creation/closure is not working
- DX OI - Alarms do not display the generated ServiceNow incidents
- DX OI - Many ServiceNow incidents and Service Alarms are being created
- DX OI - ServiceNow incidents are getting reopened many times.
- DX OI - How to assign an event to a particular group in ServiceNow based on CI's like Process, Service or URL.
- DX OI - Is it possible to create a new custom Field in ServiceNow incident that includes the UIM process name?
D) DX Dashboards
- DX Dashboard - Building a dashboard for Analyzing Browser Agent Data (White Paper)
- DX Dashboards - Scheduled reports are not sent
- DX Dashboards - Limitations
- DX Dashboards - Unable to create New dashboard, error : [in flight requests] Data too large, data for [<http_request>] would be ...], which is larger than the limit of [8537636864/7.9gb]
- DX Dashboards - Text legends and values are sometimes cut in reports
- DX Dashboards - How to query Inventory data?
- DX Dashboards - Polystat panel order by value not working
- DX Dashboards - How to use the Transforms feature, rename using regexp (a video example)
E) DX Operational Intelligence
Alarms Analytics
- DX OI - Unable to see UIM test alerts in Alarms Analytics
- DX OI - "Source" Product filter options not longer available automatically in 21.3.1
- DX OI - Auto-refresh does not work when you are looking at an Alarm Queue
- DX OI - "AlarmURL" incorrect in "Situations" view"
- DX OI - New alarms don't appear in OI Console and ServiceNow incident creation is not working
- DX OI - Unable to use the Filter option from "Alarm Analytics", it shows "Error Retrieving" message
- DX OI - Be able to reopen existing resolved Incidents for APM alerts
- DX OI - No New alarms or Alarms are not getting updated
- DX OI - Is it possible to customize the Alarm monitoring details from OI console or a config file?
- DX OI - (Enhancement) Be able to automate the way of assigning the events based on alarm type/Description/Sub description
Anomaly Detection
Performance Analytics
- DX OI - Metrics are not visible in Performance analytics and Alarms are not reported
- DX OI - How to delete unwanted metrics from Performance Analytics UI
- DX OI - New Tenants reporting ERROR "400 /oi/v3/queryTas/analytics" and "500 /oi/v2/api/Inventory/fields"
- DX OI - Performance Analytics page throws message : 500 /oi/v2/api/inventory/fields
- DX OI - Predictive Insights stop working, the page is empty
- DX OI - Unable to see CAPM Inventory data from Performance Analytics
- OI green highways not displaying correctly for custom metrics
Predictive Insight / CPA / Capacity Analytics
- DX OI - cpa-projection pod reporting CrashLoopBackOff status
- DX OI - How to create a DX Dashboards based on Prediction (capacity or performance) alarms?
- DX OI - How to export Capacity Analytics and Predictive Insight data?
Service Analytics
- DX OI - How to increase the total number of Services per tenant
- Opening the DX OI Service Analytics, APM Experience View or Map View takes almost a minute
Integration with UIM
- DX OI integration with UIM - Troubleshooting, Common Issues and Best Practices
- DX OI - UIM Alarms Custom 1 field not getting updated after UIM upgrade to 20.3.3
- DX OI - oi_connector dropping alarms - CIInformation not found for alarm
- DX OI integration with UIM - Disaster Recovery for OI Connector and APM Bridge
- DX OI integration with UIM - apm_bridge is not sending data : Could not determine UIM or TAS devices to forward for profile 0 with origin
- DX OI integration with UIM - duplicated alarms and some alarms are not cleared property in OI Console
- DX OI - UIM integration, user tags missing as part of the UIM Alarms
Integration with NetOps PM
- DX OI integration with NetOps PM and NFA - OI Connector 2.x Troubleshooting
- DX OI integration with NetOps PM and NFA - OI Connector 1.5 Troubleshooting
- DX OI - NetOps Network Flow Analysis (NFA) integration not working
- DX OI - OIConnector stops working after CAPC password changes or expires
Integration with Spectrum
- DX OI 21.3.1 - Spectrum Inventory and Services showing a wrong status, risk and health
- DX OI 21.3.1 - Spectrum Data Publisher registration fails with OIUrl FAILED
- DX OI 21.3.1 - Missing Alarms updates from Spectrum 21.2.4
- DX OI - Many alarms are cleared in Spectrum however they remain opened in Alarm Analytics
- DX OI - Spectrum 10.4.3.x or 21.x integration validation process returns Jarvis and OIUrl FAILED
- DX OI - Spectrum Data Publisher not sending inventory, error code: 503
- Spectrum topology not properly synced
- mapping Spectrum alarms to services
Integration with APM 10.7
- DX OI integration with APM 10.7 - Troubleshooting, Common Issues and Best Practices
- DX OI - Alarms explosion caused by the APM integration - How to delete All or individual APM alarms?
- DX OI - "APM Alarms do not support the action" message, unable to clear or hide any APM Alarm in OI UI
F) DX RESTMon
- AIOps RESTMon 2.2 on VM not ingesting alarms in DX OI on-prem
- RESTMon - Unable to download the latest version from the Downloads page in DX Operational Intelligence On-Premises 21.3.
G) DX Application Performance Management (APM)
APM pods
Javascript calculators
APM Configuration files
UI / Experience View / Map
- DX APM - Unable to access APM UI, error "HTTP ERROR 503 Waiting for a connection with the DB. Try again, shortly"
- DX APM - No Data in Experience View or Map due to queue length limit was reached
Traces
Upgrade
ACC Command Center
- APM ACC pods in a Crash Loop Back-off
- DX APM - ng-acc-configserver-deployment pod in CrashLoopBackOff due to FATAL: password authentication failed for user "acc"
- DX APM - Agent Download Dialog (ADD) not listing all the agent packages
Tenant data migration
H) DX App Experience Analytics (AXA)
AXA DXC ERROR c.c.a.c.p.e.impl.SpringTaskExecuter - OldProfile error detected: Profile is old for tenant App
DX AXA - No data available OR data reporting slowly
"AXA Transformer" at 400% cpu and App sessions appears but after several hours
Feedback
Yes
No
Powered by
