UMA events not reporting

Products

• DX Operational Observability
• DX Application Performance Management
• DX SaaS

Issue/Introduction

I have enabled events monitoring by updating the UMA configuration as below:

oc edit uma uma-monitor

    events:
      clamp:
        perDayLimit: 15000
      elasticDocTypeId: itoa_events_apm
      elasticDocTypeVersion: 1
      elasticProduct: ao
      enabled: true
      filter:
        fieldselector: type!=Normal
        namespace: ""
      metricsEnabled: true

however, no events are reported in dxo2

Environment

DX O2

Resolution

RECOMMENDATIONS:

1) Ensure that events feature is enabled in the cluster info pod:

kubectl exec -ti <clusterinfo-pod> -n<uma-namespace> -- sh

cd cd /tmp/clusterinfo/logs/

cat clusterInfo.log | grep -i events

env | grep -i events
clusterinfo_events_tenantToken=<token>
clusterinfo_events_elasticDocTypeId=itoa_events_apm
clusterinfo_events_perDayClamp=15000
clusterinfo_events_tenantId=
clusterinfo_events_tenantIngestionUrl=wss://apmservices-gateway.<endpoint>
clusterinfo_events_enabled=true
clusterinfo_events_namespaceFilter=
clusterinfo_events_metricsEnabled=true
clusterinfo_events_elasticProduct=ao
clusterinfo_events_elasticDocTypeVersion=1
clusterinfo_events_fieldSelector=type!=Normal

*IMPORTANT*  : If you find that clusterinfo_events_enabled=false, delete the clusterinfo-pod to force the update in UMA deployment to be taken into account

2) Check for possible connectivity, network or firewall issues, endpoints should be accessible, for example:

oc logs <clusterinfo-pod>

[EventDataReporterQueueProcessor] ERROR c.c.a.b.events.OIDataPublisher.send - Exception seen when ingesting the events data I/O error on POST request for "https://apmgateway/cloudproxy:port/jarvis/v2/ingestion" : Connect to https://apmgateway/cloudproxy:port [apmgateway/cloudproxy/IP address failed: Connection timed out

SUGGESTION:

oc rsh pod/<clusterinfo-pod>

         Suggestion 1: curl -vk https://apmgateway/cloudproxy:port

Suggestion 2: curl -vk https://IP address

         Suggestion 3: nslookup IP address

 

3) Verify that data is getting ingested in the Elasticsearch events index:

Option 1: via Elastic pod

kubectl exec -ti elasticsearch-master-0 -n<uma-namespace> -- sh

 

curl -XGET 'http://elasticsearch-master:9200/*events_apm*/_search?pretty'

 

Option 2: via dashboards

 

.

 

If you need further assistance contact Broadcom Support and provide a copy of the UMA clusterinfo log

Additional Information

Techdocs DX Agents - UMA in Openshift