DX O2 Troubleshooting and Common Issues
Article ID: 190815
Updated On:
Products
DX Application Performance Management
DX SaaS
DX Operational Observability
Issue/Introduction
The following is a high-list of techniques and suggestions to employ when troubleshooting performance, display and configuration issues in AIOps
Environment
- DX O2
- DX APM
Resolution
Table of Contents
A) Openshift / Kubernetes / Docker / UMA
Kubernetes
- How to assign a pod to a specific k8s node?
- How to clear pods in Terminating status
- Adding entries to Pod /etc/hosts
Openshift
- Unable to open Openshift console
- Unable to login to OI console, all Openshift nodes appear in a NotReady status
- many pods in Evicted status - how can I delete these unwanted pods?
Docker Registry
B) HOTFIXES
C) DX Platform Installer
Performance
License / Telemetry
NFS
Installation
- How to check "Entropy" without using dxi-entropy-check.sh
- dxi-platform-init-pod reports errors
- Disable ingress port 80
- AIOpsManagement.sh reports that our X.509 cert is not trusted by cacerts
C) DX Platform (Tenants, Login, Postgres, Elastic, NASS, TAS)
Maintenance Windows
LDAP Authentication
- Unable to update LDAP groups from the clustermanager once the tenant is created.
- LDAP Tenant authentication issues and limitations
Login issues
- Unable to login, postgresql pod in CrashLoopBackOff
- Unable to login after installation, dxi-platform-init-pod reports errors
- Unable to login due to PODs connectivity issues
- Unable to login - How to reset "masteradmin" password if forgotten
- Unable to login due to USER PASSWORD IS LOCKED
- Unable to login due to password expiration
- How to change the default password expiration policy of 180 days?
- Unable to login, "502 Bad Gateway" error after adding a 2nd nic
- Unable to login due to postgres database corruption
- Not able to login to the DX operational intelligence due to network issues
- Unable to access to DX OI/APM
Jarvis (kafka, zookeeper, kron)
- Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
- How to check Jarvis services Health
- How to check Elastic Health
- How to check the Zookeeper to Kafka connectivity
- How to check Kafka Health
- How to restart Jarvis services?
- kafka data consuming all disk space in Elastic nodes
- kafka crashloop with cluster.id mismatch in meta.properties
ElasticSearch
Postgres
- How to list databases in postgres?
- How to get the # of logged in users in DX Platform
- Unable to login due to postgres database corruption
NAS
REST API
Notifications (Emails, Ticketing, Webhooks)
ITSM Manual Grouping of Alarms
ServiceNow and NIM
- ServiceNow Integration Troubleshooting
- Nim Troubleshooting
- How to create a NIM route to access to the NIM UI?
- ServiceNow Automatic incident creation NOT working however manual creation is working
- Unable to create ServiceNow incidents manually
- ServiceNow Automatic incident creation/closure is not working
- Alarms do not display the generated ServiceNow incidents
D) DX Dashboards
- Best Practices and Design Principles for Building and Utilizing a DX Dashboard
- How to control the display of "Metric gaps" in dashboards
- Scheduled reports are not sent
- Unable to create New dashboard, error : [in flight requests] Data too large, data for [<http_request>] would be ...], which is larger than the limit of [8537636864/7.9gb]
- Text legends and values are sometimes cut in reports
- How to query Inventory data?
- How to use the Transforms feature, rename using regexp (a video example)
E) Services
F) Alarms
All Alarms
- "Sharing of Alarm Queues / Filters" Troubleshooting
- Alarms are auto closing, how to extend the period?
Situations
Anomaly Detection
G) Triage
Triage Metrics by Inventory / Performance Analytics
- No data for some UIM metrics, how to identify if the issue is related to the OIConnector or an issue in DX O2?
- No data for some UIM metrics that has Spanish characters
- How to control the display of "Metric gaps" in Performance Analytics
- Metrics are not visible in Performance analytics and Alarms are not reported
H) Predictions
Insight / Capacity
I) Integrations
NetOps
- Integration with NetOps Troubleshooting
- NetOps OIConnector requirements to include metrics from VNA-managed network objects
- NetOps OIConnector - Failed ingesting inventory (iteration 1 of 1) to TAS for CAPC tenant id Error: 500: {"error":{"code":301,"message":"DATABASE_ERROR"
- NetOps OIConnector - Repetitive message in the NetOps OConnector log : "CAPC version is not property set - assuming version 3.7.12"
- NetOps OIConnector [ERROR] GroupTreeResponse - JsonProcessingException encountered when mapping response to GroupTree.
- NetOps OIConnector - OI Connector Does Not Properly Escape Characters in tenants.properties
- NetOps OI Connector Triggers Kafka errors related to Unused Syslog Event Integration in Secured Kafka
- DX OI - OIConnector stops working after CAPC password changes or expires
UIM
- No data for some UIM metrics. How to identify if the issue is related to the UIM OIConnector or DX OI?
- No data for some UIM metrics that has Spanish characters
J) DX Application Performance Management (APM)
APM Java Agents
APM .NET Agents
APM UMA
APM Configuration files
UI / Experience View / Map
Upgrade
ACC Command Center
- APM ACC pods in a Crash Loop Back-off
- DX APM - ng-acc-configserver-deployment pod in CrashLoopBackOff due to FATAL: password authentication failed for user "acc"
- DX APM - Agent Download Dialog (ADD) not listing all the agent packages
K) DX App Experience Analytics (AXA)
- DX AXA - App sessions are not being reported, How to increase the AXA Transformer limit of 5 instances?
- AXA DXC ERROR c.c.a.c.p.e.impl.SpringTaskExecuter - OldProfile error detected: Profile is old for tenant App
- DX AXA - No data available OR data reporting slowly
- "AXA Transformer" at 400% cpu and App sessions appears but after several hours
Feedback
Yes
No
Powered by
