PathAnalyzer or PolicyManager services not healthy in VCF Operations for Networks Platform node due to OutOfMemoryError

Products

• VCF Operations for Networks

Issue/Introduction

• In Settings > Infrastructure and Support, the System Health section reports that "One or more essential services are not healthy."
  
  
• The detail of the problem alert may show the PathAnalyzer, PolicyManager services are not healthy:
  

  Operations for Networks System (aria-networks-platform)
  One or more essential services are not healthy.

  Resolution: PathAnalyzer,PolicyManager services are not healthy on aria-networks-
  platform(##.##.##.##). If the situation persists for more than 6 hours, contact
  VMware customer support.

• Check system health (./run_all.sh sudo /home/ubuntu/check-service-health.sh -p -d) for the Platform node in question reveals that the PathAnalyzer and PolicyManager services are running but not healthy.
  
  ElasticSearch is running and healthy.
HRegionServer is running and healthy.
Kafka is running and healthy.
NodeManager is running and healthy.
SaasListener is running
Restapilayer is running and healthy.
TSDB is running
DataNode is running and healthy.
Launcher is running
VIPService is running and healthy.
DatabusGateway is running and healthy.
FlinkContainer is running and healthy.
PathAnalyzer is running but not healthy.
Nginx is running and healthy.
ExpressJSApp is running
NTPSEC is running and healthy.
FoundationDB is running and healthy.
PolicyManager is running but not healthy.
All required services are running.
  
  
  
• Log review of the path analyzer log (path-analyzer.STDOUT-<timedatestamp>.log) for the Platform node in question reveals a "java.lang.OutOfMemoryError: Java heap space" error, as seen in the example below:
  
  

  2026-04-08 16:08:31 platform4 36570 0 VF : [6-Information] Running cleanup for snapshot MK 18741:2000:3624918459545986472
2026-04-08 16:08:31 platform4 36570 0 VF : [6-Information] Cleanup complete for snapshot MK 18741:2000:3624918459545986472
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /var/lib/heap-dumps/path-analyzer/java_pid36570.hprof ...
Heap dump file created [547561597 bytes in 1.424 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space
2026-04-08T16:09:07.335Z INFO veriflow.pathanalyzer.PathAnalyzerServiceImpl grpc-default-executor-78 getLatestSnapshot:238 

   

• Log review of the policy manager error log (policy-manager.STDOUT-<timedatestamp>.log.error) for the Platform node in question reveals a "StatusCode.UNAVAILABLE" status with the detail of "failed to connect to all addresses" as seen in the example below:

  2026-04-21 18:37:58,240 ERROR: MainProcess[2978] vflog.py::policy-manager::_log_helper - <_InactiveRpcError of RPC that terminated with:
 status = StatusCode.UNAVAILABLE
 details = "failed to connect to all addresses"
 debug_error_string = "{"created":"@1776796678.239836481","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3128,"referenced_errors":[{"created":"@1776796678.239835892","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
> -- {}

 

Environment

VCF Operations for Networks 6.14.1

VCF Operations for Networks 6.14.2

Cause

The OutOfMemoryError was triggered by the Path-Analyzer running the Device Modeler on a device with over 10,000 rules.

These are device rules that can be anything, not only DFW rules. These can include routes, MAC address, ARP entries etc., but we count them as rules.

 

Resolution

1. SSH into affected Platform node as the support user and switch to the ubuntu user (ub).

2. Execute the following commands to restart the affected services:

   /home/ubuntu/run_all.sh "sudo systemctl stop path-analyzer.service"

   /home/ubuntu/run_all.sh "sudo systemctl unmask path-analyzer.service"

   /home/ubuntu/run_all.sh "sudo systemctl start path-analyzer.service"

   /home/ubuntu/run_all.sh "sudo systemctl stop policy-manager.service"

   /home/ubuntu/run_all.sh "sudo systemctl unmask policy-manager.service"

   /home/ubuntu/run_all.sh "sudo systemctl start policy-manager.service"

3. Verify service health by running: ./run_all.sh sudo /home/ubuntu/check-service-health.sh -p -d

4. Observe the environment and if the issue returns, please open a Broadcom Support case for further assistance.
   
   Upload a new support bundle for the Platform node in question as well as a screenshot (you may need multiple) of the Infrastructure and Support screen as well as the details of any problem alerts.
   
   For more information, see Creating and managing Broadcom support cases.