AIOPs - Troubleshooting, Common Issues and Best Practices
Article ID: 190815
Updated On:
Products
DX Operational Intelligence
DX Application Performance Management
CA App Experience Analytics
DX APM SaaS
DX OI SaaS
DX SaaS
DX Unified Infrastructure Management (Nimsoft / UIM)
CA Spectrum
DX NetOps
Issue/Introduction
The following is a high-list of techniques and suggestions to employ when troubleshooting performance, display and configuration issues
A) Openshift / Kubernetes
B) DX Platform Installer
C) Data Platform (Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)
D) DX Dashboards
E) DX Operational Intelligence
F) DX RESTMon
G) DX Application Performance Management
H) DX App Experience Analytics
Environment
DX Platform, DX APM, DX OI, DX AXA 2x releases
Resolution
Table of Contents
A) Openshift / Kubernetes / Docker
Kubernetes
- AIOPS - How to assign a pod to a specific k8s node?
- AIOps - How to clear pods in Terminating status
- AIOPs - Adding entries to Pod /etc/hosts
Openshift
- AIOps - Unable to open Openshift console
- AIOps - Unable to login to OI console, all Openshift nodes appear in a NotReady status
- AIOps - many pods in Evicted status - how can I delete these unwanted pods?
Docker Registry
B) HOTFIXES
AIOps 23.x on-premise HOTFIXES
C) DX Platform Installer
How to Install examples
Performance
License / Telemetry
- AIOPs - License calculation for DX APM and DX OI
- DX Dashboards OI Telemetry - changing frequency of Data Collection
NFS
- DX AIOPs - Unable to start the DX platform services , many pods in CrashLoopBackOff and "Permission denied" in many pod logs
- DX AIOps Installation check - How to verify the NFS version?
- DX Platform - How to verify I/O throughput for NFS without using dbench-fio.sh
- DX Platform - How to verify I/O throughput for NFS using dbench-fio.sh
- DX AIOps - Relocating NFS storage
Entropy
Installation issues
- AIOps 22.1 - Some pods remain in CrashLoopBackOff status, error message "failed to create shim task: OCI runtime create failed"
- AIOps - Installation fails with "java.lang.IllegalArgumentException: Invalid default value"
- AIOps - dxi-platform-init-pod reports errors
- AIOps - Installation fails with [ ERROR ] Pod of Deployment .. has failed to start in the given timeframe
- AIOps - Installer fails with [elasticsearch-node-validate] [ERROR] No nodes were detected in the cluster.
- AIOps - dxi-es-admin.sh fails with error "Jarvis Elasticsearch Pod is not running"
- AIOps - Installation failed due to Postgres fails to startup : could not link file "pg_wal/xlogtemp.29"to "pg_wal/00000000001" : Input/output error
- AIOps - Is it possible to secure routes or enable SSL after installation?
- AIOps - Is it possible to change the wildcard DNS entered during DX Platform installation
Post Installation
- AIOPs - How to remove ingress port 80, we need only 443 enabled?
- DX Platform - AIOpsManagement.sh reports that our X.509 cert is not trusted by cacerts
- AIOps - dxi-component-scale.sh doesn't enable the product(s) in LaunchPad
- AIOPs - dxi_update_ingress_secret.sh incorrectly reports that "Tls is not enabled"
- AIOps - How to create jarvis apis and elasticsearch endpoints that are disabled by default?
- AIOps - apmservices-metricforward pod in Error/CrashLoopBackOff status and keeps restarting
RBAC
Upgrade issues
- DX AIOps - pods in CrashLoopBackOff , error "OCI runtime create failed: runc create failed: unable to start container process"
- AIOps - oimetricpublisher service reporting status as Failure after upgrade to 21.3.1
Maintenance
C) DX Platform (Tenants, Login, Postgres, Elastic, NASS, TAS)
Maintenance Windows
LDAP Authentication
- DX OI - Unable to assign a user to an alarm, LDAP users are not populated in the "Assign to" list
- DX AIOps 23.x - Unable to update LDAP groups from the clustermanager once the tenant is created.
- AIOps - LDAP Tenant authentication issues and limitations
Onboarding (DX Admin Emails)
- DX Platform - How to update the default "DX SaaS Admin" sender name?
- AIOps - "DX Admin"-email-link not working properly. "The system is currently down."
- AIOps - "DX Admin" emails sent during user creation uses "First name" instead of "User name"
Login issues
- AIOps - Unable to login, postgresql pod in CrashLoopBackOff
- AIOps - Unable to login after installation, dxi-platform-init-pod reports errors
- AIOps - Unable to login due to PODs connectivity issues
- AIOPs - Unable to login - How to reset "masteradmin" password if forgotten
- AIOps - Unable to login due to USER PASSWORD IS LOCKED
- AIOps - Unable to login due to password expiration
- AIOps - How to change the default password expiration policy of 180 days?
- AIOps - Unable to login, "502 Bad Gateway" error after adding a 2nd nic
- AIOps - Unable to login due to postgres database corruption
- AIOPs - Not able to login to the DX operational intelligence due to network issues
Jarvis (kafka, zookeeper, kron)
- AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
- AIOps - How to check Jarvis services Health
- AIOps - How to check Elastic Health
- AIOps - How to check the Zookeeper to Kafka connectivity
- AIOps - How to check Kafka Health
- AIOps - How to check Jarvis Kron Health
- AIOps - jarvis_jmetrics_1.0 growth up continuously
- AIOps - The jarvis-kron pod keeps restarting and some elasticsearch indices are in yellow status.
- AIOps - How to create the Jarvis apis and elasticsearch endpoints in 21.x
- AIOps - How to restart Jarvis services?
- AIOps - kafka data consuming all disk space in Elastic nodes
- AIOps - After creating a New tenant jarvis-verifier keeps crashing
- AIOps - kafka crashloop with cluster.id mismatch in meta.properties
ElasticSearch
- AIOps - ElasticSearch Queries
- AIOps - Unable to query the Elastic snapshots - repository_exception "Could not read repository data because the contents of the repository do not match its expected state"
- AIOps - How to create the Jarvis apis and elasticsearch endpoints in 21.x
- AIOps - ElasticSearch disk Full - How to reduce Elastic data retention?
- AIOps - How to clear the ElasticSearch database?
Postgres
- AIOps - How to list databases in postgres?
- AIOps - How to get the # of logged in users in DX Platform
- AIOps - Unable to login due to postgres database corruption
- AIOps - Unable to login, postgresql pod in CrashLoopBackOff
NAS
REST API
Policies
AUTOMIC
ITSM Manual Grouping of Alarms
Service Desk Management (SDM)
- AIOPs - How to configure Service Desk Management(SDM) for Ticketing
- DX OI to CA SD integration : How to pass/configure additional fields to OI?
ServiceNow and NIM
- DX AIOps - ServiceNow Integration Troubleshooting
- DX AIOps - Nim Troubleshooting
- DX AIOps- How to manually recreate a tenant configuration in NIM UI?
- DX Platform - How to create a NIM route to access to the NIM UI?
- DX OI ServiceNow integration is creating multiple ServiceNow Tickets for the same alarm
- AIOps - Changes in the DefaultTicketingManagementTemplate message template are not reflected in ServiceNow
- DX OI - ServiceNow Automatic incident creation NOT working however manual creation is working
- DX OI - Unable to create ServiceNow incidents manually
- DX OI - ServiceNow Automatic incident creation/closure is not working
- DX OI - Alarms do not display the generated ServiceNow incidents
- DX OI - Many ServiceNow incidents and Service Alarms are being created
- DX OI - ServiceNow incidents are getting reopened many times.
- DX OI - How to assign an event to a particular group in ServiceNow based on CI's like Process, Service or URL.
- DX OI - Is it possible to create a new custom Field in ServiceNow incident that includes the UIM process name?
OnPrem UI
D) DX Dashboards
- DX Dashboards - Useful Links
- Best Practices and Design Principles for Building and Utilizing a DX Dashboard
- DX Dashboards - Cross Tenant dashboards limitation
- DX AIOps - How to control the display of "Metric gaps" in dashboards
- DX Dashboards - Scheduled reports are not sent
- DX Dashboards - Limitations
- DX Dashboards - Unable to create New dashboard, error : [in flight requests] Data too large, data for [<http_request>] would be ...], which is larger than the limit of [8537636864/7.9gb]
- DX Dashboards - Text legends and values are sometimes cut in reports
- DX Dashboards - How to query Inventory data?
- DX Dashboards - Polystat panel order by value not working
- DX Dashboards - How to use the Transforms feature, rename using regexp (a video example)
E) DX Operational Intelligence
Service Analytics
Alarms
- DX OI - Alarms are auto closing, how to extend the period?
- DX OI - Unable to clear UIM Alarms, error "Alarm action(s) failed to perform"
- DX OI - Unable to see UIM test alerts in Alarms Analytics
- DX OI - "Source" Product filter options not longer available automatically in 21.3.1
- DX OI - Auto-refresh does not work when you are looking at an Alarm Queue
- DX OI - "AlarmURL" incorrect in "Situations" view"
- DX OI - New alarms don't appear in OI Console and ServiceNow incident creation is not working
- DX OI - No New alarms or Alarms are not getting updated
Situations
- DX OI - You can only pin up to a maximum of 5 filters
- DX OI - "Custom Situation definition" Troubleshooting
- DX OI - "Custom Situation definition" page empty, and unable to create a Custom Situation Definition"
Anomaly Detection
Performance Analytics
- DX OI - Unable to see UIM metric data in Performance Analytics
- DX OI - No data for some UIM metrics. How to identify if the issue is related to the OIConnector or DX OI?
- DX OI - No data for some UIM metrics that has Spanish characters
- DX OI - How to control the display of "Metric gaps" in Performance Analytics
- DX OI - Metrics are not visible in Performance analytics and Alarms are not reported
- DX OI - Performance Analytics page throws message : 500 /oi/v2/api/inventory/fields
Predictive Insight / CPA / Capacity Analytics
- DX OI - Predictive Insight - Metric Clamp
- DX OI - How to create a DX Dashboards based on Prediction (capacity or performance) alarms?
- DX OI - How to export Capacity Analytics and Predictive Insight data?
Monitored Inventory
- DX OI - Be able to Export data from Monitored Inventory
- DX OI - Add an option in the "Monitored Inventory" UI to enable/disable the "Topology Compaction" feature
NetOps Integration
- DX OI - NetOps OI Connector - What's New
- NetOps OIConnector requirements to include metrics from VNA-managed network objects
- NetOps OIConnector - Failed ingesting inventory (iteration 1 of 1) to TAS for CAPC tenant id Error: 500: {"error":{"code":301,"message":"DATABASE_ERROR"
- NetOps OIConnector - Repetitive message in the NetOps OConnector log : "CAPC version is not property set - assuming version 3.7.12"
- NetOps OIConnector [ERROR] GroupTreeResponse - JsonProcessingException encountered when mapping response to GroupTree.
- NetOps OIConnector - OI Connector Does Not Properly Escape Characters in tenants.properties
- NetOps OI Connector Triggers Kafka errors related to Unused Syslog Event Integration in Secured Kafka
- DX OI integration with NetOps PM and NFA - OI Connector 2.x Troubleshooting
- DX OI - OIConnector stops working after CAPC password changes or expires
Spectrum Integration
- DX OI "Monitored Inventory" doesn't show the IP address of Spectrum devices
- DX OI - Spectrum Data Publisher not sending inventory, error code: 503
UIM Integration
- apm_bridge 1.9 - CPU over utilization
- DX UIM apm_bridge probe is not sending the correct inventory to DX OI
- DX OI - No data for some UIM metrics. How to identify if the issue is related to the OIConnector or DX OI?
- DX OI - No data for some UIM metrics that has Spanish characters
- DX OI integration with UIM - apm_bridge error : getAllGroups: Failure reading groups from UIM com.fasterxml.jackson.core.JsonParseException
F) DX RESTMon
- DX OI - How to download RESTMon in onpremise?
- AIOps RESTMon 2.2 on VM not ingesting alarms in DX OI on-prem
- DX Restmon - ThousandEyes undefined alarms
G) DX Application Performance Management (APM)
APM Agents
- APM Introscope .Java Agent - Troubleshooting and Best Practices
- APM -How to list the versions of all the APM agents in DX APM 2.x or SaaS?
- How to configure APM Agents 2x to connect to APM EM 10.7/10.8 On-premise?
APM pods
Javascript calculators
APM Configuration files
UI / Experience View / Map
- DX APM - Unable to access APM UI, error "HTTP ERROR 503 Waiting for a connection with the DB. Try again, shortly"
- DX APM - No Data in Experience View or Map due to queue length limit was reached
Traces
Upgrade
ACC Command Center
- DX APM Command Center - How to fix corrupted DB data files
- APM ACC pods in a Crash Loop Back-off
- DX APM - ng-acc-configserver-deployment pod in CrashLoopBackOff due to FATAL: password authentication failed for user "acc"
- DX APM - Agent Download Dialog (ADD) not listing all the agent packages
Tenant data migration
H) DX App Experience Analytics (AXA)
- DX AXA - App sessions are not being reported, How to increase the AXA Transformer limit of 5 instances?
- AXA DXC ERROR c.c.a.c.p.e.impl.SpringTaskExecuter - OldProfile error detected: Profile is old for tenant App
- DX AXA - No data available OR data reporting slowly
- "AXA Transformer" at 400% cpu and App sessions appears but after several hours
Additional Information
Feedback
Yes
No
Powered by
