DX OI - New Tenants reporting ERROR "400 /oi/v3/queryTas/analytics" and "500 /oi/v2/api/Inventory/fields"

book

Article ID: 217736

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

Symptoms:

- The problem started after creating New tenants

- New tenants information doesn't appear in Elastic:

http(s)://<elastic-endpoint>/ao_dxi_tenants_1_1/_search?size=200&pretty
http(s)://<elastic-endpoint>/ao_tenants_1_1/_search?size=200&pretty

- All pods are running (verification done using: kubectl get pods -n<your-namespace>)

- No issue with memory or disk space (verification done using kubectl describe nodes)

- When opening "Service Analytics" > create a new service, the below error is reported: "400 /oi/v3/queryTas/analytics

- when opening "Performance Analytics" , the below error is reported : "500 /oi/v2/api/Inventory/fields

- restarting all services using <dx-platform-HOME>/tools/dx-admin.sh stop and start doesn't hep.

Cause

TROUBLESHOOTING
 
Focus the troubleshooting in Jarvis using below main reference:
 
DX Platform - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
https://knowledge.broadcom.com/external/article/189119
 
 
1) Checked for a possible LAG in jarvs_indexer
 
/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group jarvs_indexer

Result: confirmed there was a LAG on UIM CIs ingestion.

 
2) Checked for a possible LAG in verifier
 
/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group verifier

Result: confirmed there was a LAG and consumers kept disconnecting.
 
 
 
ROOT CAUSE:  The inventory ingestion (both CI and devices) should be completely disabled from UIM Connector since inventory/topology data should be sent to TAS only (through apmbridge) for 20.2 onwards.  

From: https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/ca-unified-infrastructure-management-probes/GA/alphabetical-probe-articles/oi-connector-ca-digital-operational-intelligence-gateway/oi-connector-ca-digital-operational-intelligence-gateway-release-notes.html#concept.dita_c00d3ea6aac2fdf0cdb9a39396f9e72b5f096057_UpgradeConsiderations

"While upgrading to oi_connector 1.38 or later versions, check and update the key subscribe_to_uim_inventory_ci value to 'no' in the oi_connector.cfg file. The key subscribe_to_uim_inventory_ci can be found in the setup section of the oi_connector.cfg file /setup/subscribe_to_uim_inventory_ci, below default value:

Environment

DX Operational Intelligence 20.2.1

 

Resolution

1. Go to UIM oi_connector raw configuration, set /setup/subscribe_to_uim_inventory_ci = no
 
2. Reset CAA_unverified_p1 offset as below
 
sh /opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --group verifier --topic CAA_unverified_p1 --reset-offsets --to-latest --execute
 
IMPORTANT: This command will clear all existing message in the kaka queue/cache.
 
3. (Optional) You might consider to increase jarvis-verifier replicas to 2 or 4 depending on your requirements, here is an example:
 
kubectl scale --replicas=4 deployment javis-verifier -n<namespace>
 
 

Additional Information

DX AIOPs Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815

Attachments