Jetstream-AIOPS Integration for spark applications

Products

• DX Application Performance Management

Issue/Introduction

Jetstream pipeline uses spark streaming jobs on yarn. We are able to collect metrics but we need your guidance on how to publish those metrics to AIOPS to view them in the dashboard and set up alerts/monitoring?

Environment

Release : 21.3

Resolution

Engineering:

From the Webex recording, my understanding of the ask is to report metrics based on Spark Client output?

Sounds matching functionality of IA LogMonitor extension. 

Basically you can use the extension to parse either a log file or console output of any command and retrieve metric data via matching patterns.

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/log-monitoring.html

command (optional): 
If the value of sourceFile is "command", then the application uses this value to run a shell command and use the output as the source for the LogMonitor (for example, vmstat).

====

For JMX metrics, you can use IA Remote JMX Monitoring:

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/Remote-JMX-Monitoring.html

 

However you need to specify the port# in configuration. I don't think we will "scan" and find dynamic ports in a range.