Report Advisor stuck while extracting reports; unable to run reports in Report Advisor

Products

VMware Smart Assurance

Issue/Introduction

Unable to run reports in Report Advisor. Rebooting server solves the issue for a short time, but the issue returns.

catalina.out:

05-May-2016 17:41:01.884 INFO [http-nio-8443-exec-20] org.apache.coyote.http11.AbstractHttp11Processor.process Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid9664.hprof ...
Dump file is incomplete: file size limit
06-May-2016 10:34:29.851 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: GC overhead limit exceeded
               at java.util.TreeMap.put(TreeMap.java:577)
               at java.util.TreeSet.add(TreeSet.java:255)
               at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
               at java.util.TreeSet.addAll(TreeSet.java:312)


Voyence.log:

WARN : 2016-05-06 17:09:09,475: SchedulerServiceImpl: Job finished executing: jobId=10136
ERROR: 2016-05-06 17:15:12,985: DiskPageStore$SessionEntry: Error writing to a channel sun.nio.ch.FileChannelImpl@14da691
java.io.IOException: No space left on device
               at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
               at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
               at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
               at sun.nio.ch.IOUtil.write(IOUtil.java:65)
               at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
               at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
               at org.apache.wicket.protocol.http.pagestore.DiskPageStore$SessionEntry.savePage(DiskPageStore.java:246)
               at org.apache.wicket.protocol.http.pagestore.DiskPageStore.flushPagesToSaveList(DiskPageStore.java:895)
               at org.apache.wicket.protocol.http.pagestore.DiskPageStore$PageSavingThread.run(DiskPageStore.java:965)

Environment

NCM - 9.x

Resolution

Clear out old logs in the $TOMCAT_HOME/logs directory. Generally, 30 days of logs is sufficient for troubleshooting issues in RA.
Increase the max memory in .bash_profile (change to -Xmx=4096m) located in $TOMCAT_HOME
Restart tomcat service for changes to take effect.

Additional Information

GC overhead limit exceeded refers to the Garbage Collector. When an object on the heap is no longer usable, it is marked for garbage collection. If/when your application is running low on memory the garbage collector will run, throwing out unreachable objects and freeing up space so it can be re-used. Garbage collection errors indicate that the JVM took too long to free up memory during its GC process.

The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small.

This kind of OutOfMemoryError can be caused if user requests the available resources in the JVM. When this occurs, performance will degrade aggressively.

Things to check

Check for .hprof files in /var/cores or in $VOYENCE_HOME/dump/. User can also run locate *.hprof to find and heap files. User need to move these off of the server (or delete them) as they consume huge space. The most recent dump file need to be shared with support along with catalina.out and Voyence.log.
Check the file space for RA to be sure that server is not out of space (No space left on device). Space is usually consumed by logs or hprof files. Logs are stored in $TOMCAT_HOME/logs.