AIOPs Troubleshooting and Common Issues
Article ID: 190815
Updated On: 01-17-2025
Products
DX Operational Intelligence
DX Application Performance Management
CA App Experience Analytics
DX APM SaaS
DX OI SaaS
DX SaaS
DX Unified Infrastructure Management (Nimsoft / UIM)
Spectrum
Network Observability
DX Operational Observability
Issue/Introduction
The following is a high-list of techniques and suggestions to employ when troubleshooting performance, display and configuration issues in AIOps
Environment
- DX O2
- DX Platform
- DX APM
- DX OI
- DX AXA
Resolution
Table of Contents
- A) Openshift / Kubernetes / Docker
- B) HOTFIXES
- C) DX Platform Installer
- C) DX Platform (Tenants, Login, Postgres, Elastic, NASS, TAS)
- D) DX Dashboards
- E) Services
- F) Alarms
- G) Triage
- H) Predictions
- I) Inventory
- J) Integrations
- K) DX RESTMon
- L) DX Application Performance Management (APM)
- M) DX App Experience Analytics (AXA)
A) Openshift / Kubernetes / Docker
Kubernetes
- AIOPS - How to assign a pod to a specific k8s node?
- AIOps - How to clear pods in Terminating status
- AIOPs - Adding entries to Pod /etc/hosts
Openshift
- AIOps - Unable to open Openshift console
- AIOps - Unable to login to OI console, all Openshift nodes appear in a NotReady status
- AIOps - many pods in Evicted status - how can I delete these unwanted pods?
Docker Registry
B) HOTFIXES
C) DX Platform Installer
Performance
License / Telemetry
- AIOPs - License calculation for DX APM and DX OI
- DX Dashboards OI Telemetry - changing frequency of Data Collection
NFS
- DX AIOPs - Unable to start the DX platform services , many pods in CrashLoopBackOff and "Permission denied" in many pod logs
- DX AIOps Installation check - How to verify the NFS version?
- DX Platform - How to verify I/O throughput for NFS without using dbench-fio.sh
- DX Platform - How to verify I/O throughput for NFS using dbench-fio.sh
- DX AIOps - Relocating NFS storage
Entropy
Installation issues
- AIOps - dxi-platform-init-pod reports errors
- AIOps - Installation failed due to Postgres fails to startup : could not link file "pg_wal/xlogtemp.29"to "pg_wal/00000000001" : Input/output error
Post Installation
- AIOPs - How to remove ingress port 80, we need only 443 enabled?
- DX Platform - AIOpsManagement.sh reports that our X.509 cert is not trusted by cacerts
- AIOps - How to create jarvis apis and elasticsearch endpoints that are disabled by default?
RBAC
C) DX Platform (Tenants, Login, Postgres, Elastic, NASS, TAS)
Maintenance Windows
LDAP Authentication
- DX OI - Unable to assign a user to an alarm, LDAP users are not populated in the "Assign to" list
- DX AIOps 23.x - Unable to update LDAP groups from the clustermanager once the tenant is created.
- AIOps - LDAP Tenant authentication issues and limitations
Onboarding (DX Admin Emails)
- DX Platform - How to update the default "DX SaaS Admin" sender name?
- AIOps - "DX Admin"-email-link not working properly. "The system is currently down."
- AIOps - "DX Admin" emails sent during user creation uses "First name" instead of "User name"
Login issues
- AIOps - Unable to login, postgresql pod in CrashLoopBackOff
- AIOps - Unable to login after installation, dxi-platform-init-pod reports errors
- AIOps - Unable to login due to PODs connectivity issues
- AIOPs - Unable to login - How to reset "masteradmin" password if forgotten
- AIOps - Unable to login due to USER PASSWORD IS LOCKED
- AIOps - Unable to login due to password expiration
- AIOps - How to change the default password expiration policy of 180 days?
- AIOps - Unable to login, "502 Bad Gateway" error after adding a 2nd nic
- AIOps - Unable to login due to postgres database corruption
- AIOPs - Not able to login to the DX operational intelligence due to network issues
- AIOps - Unable to access to DX OI/APM
Jarvis (kafka, zookeeper, kron)
- AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
- AIOps - How to check Jarvis services Health
- AIOps - How to check Elastic Health
- AIOps - How to check the Zookeeper to Kafka connectivity
- AIOps - How to check Kafka Health
- AIOps - How to check Jarvis Kron Health
- AIOps - jarvis_jmetrics_1.0 growth up continuously
- AIOps - The jarvis-kron pod keeps restarting and some elasticsearch indices are in yellow status.
- AIOps - How to create the Jarvis apis and elasticsearch endpoints in 21.x
- AIOps - How to restart Jarvis services?
- AIOps - kafka data consuming all disk space in Elastic nodes
- AIOps - After creating a New tenant jarvis-verifier keeps crashing
- AIOps - kafka crashloop with cluster.id mismatch in meta.properties
ElasticSearch
- AIOps - ElasticSearch Queries
- AIOps - Unable to query the Elastic snapshots - repository_exception "Could not read repository data because the contents of the repository do not match its expected state"
- AIOps - How to create the Jarvis apis and elasticsearch endpoints in 21.x
- AIOps - ElasticSearch disk Full - How to reduce Elastic data retention?
Postgres
- AIOps - How to list databases in postgres?
- AIOps - How to get the # of logged in users in DX Platform
- AIOps - Unable to login due to postgres database corruption
- AIOps - Unable to login, postgresql pod in CrashLoopBackOff
NAS
REST API
Policies
Notifications (Emails, Ticketing, Webhooks)
Emails
ITSM Manual Grouping of Alarms
Service Desk Management (SDM)
ServiceNow and NIM
- DX AIOps - ServiceNow Integration Troubleshooting
- DX AIOps - Nim Troubleshooting
- DX AIOps- How to manually recreate a tenant configuration in NIM UI?
- DX Platform - How to create a NIM route to access to the NIM UI?
- DX OI ServiceNow integration is creating multiple ServiceNow Tickets for the same alarm
- DX OI - ServiceNow Automatic incident creation NOT working however manual creation is working
- DX OI - Unable to create ServiceNow incidents manually
- DX OI - ServiceNow Automatic incident creation/closure is not working
- DX OI - Alarms do not display the generated ServiceNow incidents
OnPrem UI
D) DX Dashboards
- DX Dashboards - Useful Links
- Best Practices and Design Principles for Building and Utilizing a DX Dashboard
- DX Dashboards - Cross Tenant dashboards limitation
- DX AIOps - How to control the display of "Metric gaps" in dashboards
- DX Dashboards - Scheduled reports are not sent
- DX Dashboards - Unable to create New dashboard, error : [in flight requests] Data too large, data for [<http_request>] would be ...], which is larger than the limit of [8537636864/7.9gb]
- DX Dashboards - Text legends and values are sometimes cut in reports
- DX Dashboards - How to query Inventory data?
- DX Dashboards - How to use the Transforms feature, rename using regexp (a video example)
E) Services
F) Alarms
All Alarms
- DX OI - "Sharing of Alarm Queues / Filters" Troubleshooting
- DX OI - Alarms are auto closing, how to extend the period?
- DX OI - Unable to clear UIM Alarms, error "Alarm action(s) failed to perform"
- DX OI - Unable to see UIM test alerts in Alarms Analytics
- DX OI - New alarms don't appear in OI Console and ServiceNow incident creation is not working
- DX OI - No New alarms or Alarms are not getting updated
Situations
- DX OI - "Custom Situation definition" Troubleshooting
- DX OI - "Custom Situation definition" page empty, and unable to create a Custom Situation Definition"
Anomaly Detection
G) Triage
Triage Metrics by Inventory / Performance Analytics
- DX OI - Unable to see UIM metric data in Performance Analytics
- DX OI - No data for some UIM metrics. How to identify if the issue is related to the OIConnector or DX OI?
- DX OI - No data for some UIM metrics that has Spanish characters
- DX OI - How to control the display of "Metric gaps" in Performance Analytics
- DX OI - Metrics are not visible in Performance analytics and Alarms are not reported
H) Predictions
Insight / Capacity
I) Inventory
J) Integrations
NetOps
- DX OI - Integration with NetOps Troubleshooting
- DX OI - NetOps OI Connector - What's New
- NetOps OIConnector requirements to include metrics from VNA-managed network objects
- NetOps OIConnector - Failed ingesting inventory (iteration 1 of 1) to TAS for CAPC tenant id Error: 500: {"error":{"code":301,"message":"DATABASE_ERROR"
- NetOps OIConnector - Repetitive message in the NetOps OConnector log : "CAPC version is not property set - assuming version 3.7.12"
- NetOps OIConnector [ERROR] GroupTreeResponse - JsonProcessingException encountered when mapping response to GroupTree.
- NetOps OIConnector - OI Connector Does Not Properly Escape Characters in tenants.properties
- NetOps OI Connector Triggers Kafka errors related to Unused Syslog Event Integration in Secured Kafka
- DX OI - OIConnector stops working after CAPC password changes or expires
Spectrum
UIM
- DX UIM apm_bridge probe is not sending the correct inventory to DX OI
- DX OI - No data for some UIM metrics. How to identify if the issue is related to the OIConnector or DX OI?
- DX OI - No data for some UIM metrics that has Spanish characters
K) DX RESTMon
- DX OI - How to download RESTMon in onpremise?
- AIOps RESTMon 2.2 on VM not ingesting alarms in DX OI on-prem
L) DX Application Performance Management (APM)
APM Agents
- APM Introscope .Java Agent - Troubleshooting and Best Practices
- APM -How to list the versions of all the APM agents in DX APM 2.x or SaaS?
- How to configure APM Agents 2x to connect to APM EM 10.7/10.8 On-premise?
APM Configuration files
UI / Experience View / Map
Traces
Upgrade
ACC Command Center
- DX APM Command Center - How to fix corrupted DB data files
- APM ACC pods in a Crash Loop Back-off
- DX APM - ng-acc-configserver-deployment pod in CrashLoopBackOff due to FATAL: password authentication failed for user "acc"
- DX APM - Agent Download Dialog (ADD) not listing all the agent packages
M) DX App Experience Analytics (AXA)
- DX AXA - App sessions are not being reported, How to increase the AXA Transformer limit of 5 instances?
- AXA DXC ERROR c.c.a.c.p.e.impl.SpringTaskExecuter - OldProfile error detected: Profile is old for tenant App
- DX AXA - No data available OR data reporting slowly
- "AXA Transformer" at 400% cpu and App sessions appears but after several hours
Feedback
Yes
No
Powered by
