DX OI - New Tenants reporting ERROR "400 /oi/v3/queryTas/analytics" and "500 /oi/v2/api/Inventory/fields"


Article ID: 217736


Updated On:


DX Operational Intelligence



- The problem started after creating New tenants

- New tenants information doesn't appear in Elastic:


- All pods are running (verification done using: kubectl get pods -n<your-namespace>)

- No issue with memory or disk space (verification done using kubectl describe nodes)

- When opening "Service Analytics" > create a new service, the below error is reported: "400 /oi/v3/queryTas/analytics

- when opening "Performance Analytics" , the below error is reported : "500 /oi/v2/api/Inventory/fields

- restarting all services using <dx-platform-HOME>/tools/dx-admin.sh stop and start doesn't hep.


Focus the troubleshooting in Jarvis using below main reference:
DX Platform - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
1) Checked for a possible LAG in jarvs_indexer
/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group jarvs_indexer

Result: confirmed there was a LAG on UIM CIs ingestion.

2) Checked for a possible LAG in verifier
/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group verifier

Result: confirmed there was a LAG and consumers kept disconnecting.
ROOT CAUSE:  The inventory ingestion (both CI and devices) should be completely disabled from UIM Connector since inventory/topology data should be sent to TAS only (through apmbridge) for 20.2 onwards.  

From: https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/ca-unified-infrastructure-management-probes/GA/alphabetical-probe-articles/oi-connector-ca-digital-operational-intelligence-gateway/oi-connector-ca-digital-operational-intelligence-gateway-release-notes.html#concept.dita_c00d3ea6aac2fdf0cdb9a39396f9e72b5f096057_UpgradeConsiderations

"While upgrading to oi_connector 1.38 or later versions, check and update the key subscribe_to_uim_inventory_ci value to 'no' in the oi_connector.cfg file. The key subscribe_to_uim_inventory_ci can be found in the setup section of the oi_connector.cfg file /setup/subscribe_to_uim_inventory_ci, below default value:


DX Operational Intelligence 20.2.1



1. Go to UIM oi_connector raw configuration, set /setup/subscribe_to_uim_inventory_ci = no
2. Reset CAA_unverified_p1 offset as below
sh /opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --group verifier --topic CAA_unverified_p1 --reset-offsets --to-latest --execute
IMPORTANT: This command will clear all existing message in the kaka queue/cache.
3. (Optional) You might consider to increase jarvis-verifier replicas to 2 or 4 depending on your requirements, here is an example:
kubectl scale --replicas=4 deployment javis-verifier -n<namespace>

Additional Information

DX AIOPs Troubleshooting, Common Issues and Best Practices