AIOPs - Troubleshooting, Common Issues and Best Practices

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

The following is a high-list of techniques and suggestions to employ when troubleshooting performance, display and configuration issues

A) Openshift / Kubernetes
B) DX Platform Installer
C) Data Platform (Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)
D) DX Dashboards
E) DX Operational Intelligence
F) DX Application Performance Management

 

Environment

DX Platform Installer 20.x and higher
DX Operational Intelligence 20.x and higher
DX Application Performance Management 20.x and higher
DX AXA 20.x and higher

Resolution

Table of Contents

• Common Issues
  • A) Openshift / Kubernetes
  • B) DX Platform Installer
    • NFS
    • Sizing Calculator
    • DX Platform Installer
    • HOTFIX
    • Installation issues
    • Post Installation
    • Upgrade issues
    • Deploy New Images
    • Maintenance
    • Vulnerabilities
  • C) DX Platform (Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)
    • Login issues
    • Jarvis (kafka, zookeeper)
    • ElasticSearch
    • REST API
    • Notifications (Emails, Webhooks, Ticketing) 
    • Cluster Management
  • D) DX Dashboards
  • E) DX Operational Intelligence 
    • Alarms Analytics
    • Situations
    • Performance Analytics
    • DSP / DSP Integrator / Anomaly Detection
    • Predictive Insight / CPA / Capacity Analytics
    • Service Analytics
    • Integrations
    • Integration with UIM
    • Integration with NetOps PM and NFA
    • Integration with Spectrum
    • Integration with APM 10.7
  • F) DX Application Performance Management (APM)
    • Experience View / Map
    • Traces Tab
    • Upgrade
    • ACC Command Center
    • Tenant data migration
  •  
• Compatibility Guides
  • • DX OI
    • DX APM
  •  
• Youtube Playlists
  • • DX OI 
    • DX OI integrations
    • DX Dashboards
    • DX APM
    • DX AXA
    • DX NEtOps
    • Automic
  •  
• AIOps Notifications

Common Issues

 

A) Openshift / Kubernetes

 

• AIOps - Unable to open Openshift console
• AIOps - Unable to login to OI console, all Openshift nodes appear in a NotReady status 
• AIOps - many pods in Evicted status - how can I delete these unwanted pods?

 

B) DX Platform Installer

NFS

• AIOps - How to verify throughput for NFS (IOPs and Speed)

 

Sizing Calculator

• AIOps - High elasticsearch nodes requirement (> 10 elastic nodes). Is this correct?

 

DX Platform Installer

21.3.1:

• • Example how to install Openshift 3.11 + DX Platform 21.3.1 OI and APM (9 x 32 GB Mem servers ) + 3 ElasticNodes NFS DataStorage + ClusterAdministrator
  • Example how to install Kubernetes + DX Platform 21.3.1 OI and APM (5 x 64GB Mem servers ) + 3 ElasticNodes NFS DataStorage + ClusterAdministrator

20.2.1:

• • Example how to install Openshift 3.11 + DX Platform 20.2.1 OI+APM+AXA (template 4 servers x 64GB Mem)
  • Example how to Install Openshift 3.11 + DX Platform 20.2.1 OI+APM (template 6 servers x 64GB Mem) + 3 ElasticNodes
  • Example how of Install Kubernetes + DX Platform 20.2.1 OI+APM+AXA (template 5 servers x 64GB Mem)

 

HOTFIX

• Example How to install DX Platform 21.3.1 HOTIFIX 1

 

Installation issues

• AIOps - Installation fails with "java.lang.IllegalArgumentException: Invalid default value"
• AIOps - dxi-platform-init-pod reports errors
• AIOps - Installation fails with [ ERROR ] Pod of Deployment .. has failed to start in the given timeframe
• AIOps - Installer fails with [elasticsearch-node-validate] [ERROR] No nodes were detected in the cluster.
• AIOps - Unable to activate DX OI, APM or AXA using dxi-component-scale.sh
• AIOps - dxi-es-admin.sh fails with error "Jarvis Elasticsearch Pod is not running"
• AIOps - Is it possible to secure routes or enable SSL after installation?
• AIOps - Is it possible to change the wildcard DNS entered during DX Platform installation

 

Post Installation

• AIOps - How to enable High Availability for ElasticSearch?
• AIOps - How to create jarvis apis and elasticsearch endpoints that are disabled by default?

 

Upgrade issues

• AIOps - oimetricpublisher service reporting status as Failure after upgrade to 21.3.1

 

Deploy New Images

• AIOps - How to apply a new hotfix image?

 

Maintenance

• AIOps - How to gracefully restart all DX services and servers?
• AIOps - what unnecessary logs and files can I delete?

 

Vulnerabilities

• DX OI 20.2.1, 21.3.1 - Security Advisory for Log4j 2 CVE-2021-44228 Vulnerability 
• AIOps - 20.2 Vulnerability issues

 

C) DX Platform (Cluster Management, Login, Postgres, Jarvis, Elastic, NASS, TAS)

Login issues

• AIOps 21.3.1 - Unable to login to newly created Tenants, the login process enter in a loop if the top domain name is ".local"
• AIOps - Unable to login after installation, dxi-platform-init-pod reports errors
• AIOps - Unable to login due to PODs connectivity issues
• AIOPs - Unable to login - How to reset "masteradmin" password if forgotten
• AIOps - Unable to login due to USER PASSWORD IS LOCKED
• AIOps - Unable to login due to password expiration
• AIOps - How to change the default password expiration policy of 180 days?
• AIOps - Unable to login, "502 Bad Gateway" error after adding a 2nd nic
• AIOps 20.2.1 - Unable to login, login is successful but it immediately logout
• AIOps - Unable to login due to postgres database corruption

Jarvis (kafka, zookeeper)

• AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
• AIOps - How to restart Jarvis services?
• AIOps - kafka data consuming all disk space in Elastic nodes
• AIOps - After creating a New tenant jarvis-verifier keeps crashing

 

ElasticSearch

• AIOps - ElasticSearch Queries
• AIOps -  ElasticSearch disk Full - How to reduce Elastic data retention?
• AIOps - How to clear the ElasticSearch database?

 

REST API

• AIOps - Examples of how to query for Alarms in SaaS?
• AIOps - Examples of how to query for Alarms, Metrics and Topology using REST APIs?

 

 

Notifications (Emails, Webhooks, Ticketing) 

• DX AIOps - ServiceNow Integration Troubleshooting
• DX AIOps - Nim Troubleshooting
• DX AIOps- How to manually recreate a tenant configuration in NIM UI?
• DX Platform - How to create a NIM route to access to the NIM UI?
  
  
• AIOps - Do we support HTML for alarm email notification?
•  
• DX Platform - Unable to open Email Channel, instead we get an empty page
• DX OI - Be able to reopen existing resolved Incidents for APM alerts
• DX OI - Unable to create ServiceNow incidents manually
• DX OI - ServiceNow Automatic incident creation/closure is not working
• DX OI - Alarms do not display the generated ServiceNow incidents
• DX OI - Many ServiceNow incidents and Service Alarms are being created
• DX OI - ServiceNow incidents are getting reopened many times.
• DX OI - ServiceNow incidents are created but OI console still shows the “Open ticket” option rather than the incident number
• DX OI - How to assign an event to a particular group in ServiceNow based on CI's like Process, Service or URL.
• DX OI - Is it possible to create a new custom Field in ServiceNow incident that includes the UIM process name?

 

Cluster Management

• AIOps - LDAP Tenant authentication issues and limitations

 

D) DX Dashboards

 

• DX Dashboards - How to query Inventory data?
• DX Dashboards - Polystat panel order by value not working
• DX Dashboards - How to use the Transforms feature, rename using regexp (a video example)

 

E) DX Operational Intelligence 

 

Alarms Analytics

• DX OI - New alarms don't appear in OI Console and ServiceNow incident creation is not working
• DX OI - Unable to use the Filter option from "Alarm Analytics", it shows "Error Retrieving" message
• DX OI - No New alarms or Alarms are not getting updated
• DX OI - Is it possible to customize the Alarm monitoring details from OI console or a config file?
• DX OI - (Enhancement) Be able to automate the way of assigning the events based on alarm type/Description/Sub description

 

Situations

• DX OI - doi-situations-0 pod in Terminating status

 

Performance Analytics

• DX OI - Metrics are not visible in Performance analytics and Alarms are not reported
• DX OI - How to delete unwanted metrics from Performance Analytics UI
• DX OI - New Tenants reporting ERROR "400 /oi/v3/queryTas/analytics" and "500 /oi/v2/api/Inventory/fields"
• DX OI - Performance Analytics page throws message : 500 /oi/v2/api/inventory/fields
• DX OI - Predictive Insights stop working, the page is empty
• DX OI - Unable to see CAPM Inventory data from Performance Analytics
• OI green highways not displaying correctly for custom metrics

 

DSP / DSP Integrator / Anomaly Detection

• DX OI - DSP integrator is running at almost 200 % CPU and full memory usage
• DX OI - How to configure DSP to process only CAPM metrics

 

Predictive Insight / CPA / Capacity Analytics

• DX OI - cpa-projection pod reporting CrashLoopBackOff status
• DX OI - How to create a DX Dashboards based on Prediction (capacity or performance) alarms?
• DX OI - How to export Capacity Analytics and Predictive Insight data?

 

Service Analytics

• Opening the DX OI Service Analytics, APM Experience View or Map View takes almost a minute

 

Integrations

 

Integration with UIM

• • DX OI integration with UIM - Troubleshooting, Common Issues and Best Practices

• • DX OI - oi_connector dropping alarms - CIInformation not found for alarm
  • DX OI integration with UIM - Disaster Recovery for OI Connector and APM Bridge
  • DX OI integration with UIM - apm_bridge is not sending data : Could not determine UIM or TAS devices to forward for profile 0 with origin
  • DX OI integration with UIM - duplicated alarms and some alarms are not cleared property in OI Console
  • DX OI - UIM integration, user tags missing as part of the UIM Alarms

Integration with NetOps PM and NFA

• • DX OI integration with NetOps PM and NFA - OI Connector 2.x Troubleshooting
  • DX OI integration with NetOps PM and NFA - OI Connector 1.5 Troubleshooting
  • DX OI - NetOps Network Flow Analysis (NFA) integration not working
  • DX OI - OIConnector stops working after CAPC password changes or expires

Integration with Spectrum

• • DX OI 21.3.1 - Spectrum Data Publisher registration fails with OIUrl FAILED
  • DX OI 21.3.1 - Missing Alarms updates from Spectrum 21.2.4
  • DX OI - Many alarms are cleared in Spectrum however they remain opened in Alarm Analytics
  • DX OI - Spectrum 10.4.3.x or 21.x integration validation process returns Jarvis and OIUrl FAILED
  • DX OI - Spectrum Data Publisher not sending inventory, error code: 503
  • Spectrum topology not properly synced
  • mapping Spectrum alarms to services

Integration with APM 10.7

• • DX OI integration with APM 10.7 - Troubleshooting, Common Issues and Best Practices

OI 1.3.x : https://knowledge.broadcom.com/external/article/208009

• • DX OI - Alarms explosion caused by the APM integration - How to delete All or individual APM alarms?
  • DX OI - "APM Alarms do not support the action" message, unable to clear or  hide any APM Alarm in OI UI

 

 

F) DX Application Performance Management (APM)

 

Experience View / Map

• DX APM - No Data in Experience View or Map due to queue length limit was reached

 

Traces Tab

• DX APM - If ElasticSearch node#1 is shutdown "Trace tab" reports No Data

 

Upgrade

• DX APM - REST API to import and delete Management Module

 

ACC Command Center

• DX APM - ng-acc-configserver-deployment pod in CrashLoopBackOff due to FATAL: password authentication failed for user "acc"
• DX APM - ng-acc-configserver pods in CrashLoopBackOff due to ACC postgres DB corruption

 

Tenant data migration

• DX APM - Removing service data and fixing partitioning

 

Compatibility Guides

 

DX OI

https://support.broadcom.com/external/content/release-announcements/DX-Operational-Intelligence-Interoperability/18129

DX APM

https://techdocs.broadcom.com/us/product-content/status/compatibility-matrix/application-performance-management-compatibility-guide.html?r=2
https://support.broadcom.com/external/content/release-announcements/CA-APM-Release-and-Support-Lifecycle-Dates/3058

 

Youtube Playlists

 

DX OI : https://www.youtube.com/playlist?list=PLynEdQRJawmxJMNgDiLiCxa1lY8mdNGCE

DX OI integrations

https://www.youtube.com/playlist?list=PLynEdQRJawmyTGCbQOCXxpp3umLjJHtAL 

https://www.youtube.com/playlist?list=PLynEdQRJawmzPUlRsE07wibObVR4R9_3L

DX Dashboards

https://www.youtube.com/playlist?list=PLynEdQRJawmxyamCxLF8zy3L_gAM1ZlGb

https://www.youtube.com/playlist?list=PLynEdQRJawmyWfK7OOB_woXchiTSKsprH 

DX APM : https://www.youtube.com/playlist?list=PLynEdQRJawmyo3BPtHwktb5G2InPa7gMF

DX AXA : https://www.youtube.com/watch?v=Q-c0e4NiUgE&list=PLynEdQRJawmyYyuacvctTh-hIVDcCjLjD&index=9 

DX NEtOps : https://www.youtube.com/playlist?list=PLynEdQRJawmw2wb-7NevTw25xU-UN9iwp

Automic : https://www.youtube.com/playlist?list=PLynEdQRJawmynZKKqIh-aD5M2T0Lbp8Yv 

 

AIOps Notifications

https://support.broadcom.com/user/notifications.html