Security Intelligence Recommendations failure due to termination of Spark Kubernetes driver

Products

• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vDefend Firewall

Issue/Introduction

Security Intelligence Recommendations failure due to termination of Spark Kubernetes driver

UI Error "Job - Job 17 cancelled because SparkContext was shut down"

rec-driver logs shows the below details :

WARN dispatcher-CoarseGrainedScheduler TaskSetManager - Stage 32 contains a task of very large size (3384 KiB). The maximum recommended task size is 1000 KiB.

WARN task-result-getter-3 TaskSetManager - Lost task 7.0 in stage 33.0 (TID 196) (192.168.5.236 executor 2): java.lang.OutOfMemoryError: Java heap space

ERROR main RecommendationJob - Job 17 cancelled because SparkContext was shut down","kubernetes":{"pod_name":"rec-e55f10ac72859d3beb557c4ddb6b272f-driver","namespace_name":"nsxi-platform","pod_id":"9a636b17-a593-49b2-a0ae-3804976ecf5d","host":"napp-cluster-default-workers-kcpx6-5c9b575b5bxbp4cq-m5dxh","container_name":"spark-kubernetes-driver","docker_id":"d6b0327c5996155796dbd334779cd9e7a41615f1424a70896de8fd9fe2121169","container_hash":"projects.registry.vmware.com/nsx_application_platform/clustering/recommendation-spark-job@sha256:86bc62113b8c6041b7f157a30a390bf4cc396531005e695f6fccf6fe4d3204f7","container_image":"sha256:df907191e4ab93d5568129ca993366a0b52fd661c92859cd614332ebd5f24c31"}}

 

 

Environment

NAPP 4.2

Cause

The termination reason for the Spark Kubernetes driver in the provided logs is Error, specifically related to an ExecutorLostFailure caused by a JVM Out Of Memory (OOM) condition.

 

Resolution

Please contact Broadcom Support for resolving this issue