Smarts NCM: OutOfMemory exceptions seen in the jboss.log, server.log, or powerup.log; What data do I collect?

Products

• VMware

Issue/Introduction

Symptoms:


What data do I collect if I see OutOfMemory exceptions in the jboss.log, server.log, or powerup.log for Smarts Network Configuration Manager (Smarts NCM) environment?

OutOfMemory exceptions seen in the jboss.log, server.log, or powerup.log for Smarts NCM environment

Environment

VMware Smart Assurance - NCM

Resolution

If you see OutOfMemory exceptions in the jboss.log, server.log, or powerup.log for Smarts NCM environment, do the following:

1. Collect all AS logs with either of the following:
   
   ./saveLogs.pl all AP --save-rolled-logs
   
   OR
   
    /etc/voyence.conf
   ./saveLogs.pl jboss --save-rolled-logs
   
    
2. Collect a heap dump if it has been created:
   
   locate *.hprof 
   
    
3. Find the PID of jboss.Main (jboss):
   
    ps -ef | grep jboss.Main
   
    
4. Create a heap dump if one does not exist with either of the following commands:
   
    jmap -F -dump:format=b,file=forcedDump.hprof <jboss.Main pid>
   
   OR
   
     ./jmap -F -dump:format=b,file=forcedDump.hprof <jboss.Main pid>
   
    
5. Gather the results of the following commands (this command shows real-time stats):
    top (to see all processes and their usage)
    top -p <jboss.Main pid> (to see how much CPU/memory it is occupying)
   
    
6. Collect he size of the 10 largest tables in the Postgres database:
   
   su pgdba
    psql voyencedb voyence
    SELECT * FROM cm_calculate_table_sizes LIMIT 10;
    \q
    exit
    
   You may pipe the output to a file by using the following instructions (contents will not output to the screen):
    
    su pgdba
    psql voyencedb voyence
    \pset pager
    \o /tmp/top_ten_tables.txt
    \q
    exit
   
   The output will be in the /tmp directory in a file named top_ten_tables.txt.
   
    
7. Once you have collected the data from the preceding steps, open a Service Request (SR) with EMC Customer Support through the EMC Online Support site (https://www.support.emc.com) or by phone. Reference this solution ID in your SR and attach these files to your SR for review.