Smarts NCM: Report Advisor stuck while extracting reports; unable to run reports in Report Advisor
Article ID: 331160
Updated On:
Products
VMware Smart Assurance
Issue/Introduction
Symptoms:
Unable to run reports in Report Advisor. Rebooting server solves the issue for a short time, but the issue returns.
Unable to run reports in Report Advisor. Rebooting server solves the issue for a short time, but the issue returns.
Environment
VMware Smart Assurance - NCM
Cause
catalina.out:
05-May-2016 17:41:01.884 INFO [http-nio-8443-exec-20] org.apache.coyote.http11.AbstractHttp11Processor.process Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid9664.hprof ...
Dump file is incomplete: file size limit
06-May-2016 10:34:29.851 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.TreeMap.put(TreeMap.java:577)
at java.util.TreeSet.add(TreeSet.java:255)
at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
at java.util.TreeSet.addAll(TreeSet.java:312)
Voyence.log:
WARN : 2016-05-06 17:09:09,475: SchedulerServiceImpl: Job finished executing: jobId=10136
ERROR: 2016-05-06 17:15:12,985: DiskPageStore$SessionEntry: Error writing to a channel sun.nio.ch.FileChannelImpl@14da691
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
at org.apache.wicket.protocol.http.pagestore.DiskPageStore$SessionEntry.savePage(DiskPageStore.java:246)
at org.apache.wicket.protocol.http.pagestore.DiskPageStore.flushPagesToSaveList(DiskPageStore.java:895)
at org.apache.wicket.protocol.http.pagestore.DiskPageStore$PageSavingThread.run(DiskPageStore.java:965)
at java.lang.Thread.run(Thread.java:745)
05-May-2016 17:41:01.884 INFO [http-nio-8443-exec-20] org.apache.coyote.http11.AbstractHttp11Processor.process Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid9664.hprof ...
Dump file is incomplete: file size limit
06-May-2016 10:34:29.851 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.TreeMap.put(TreeMap.java:577)
at java.util.TreeSet.add(TreeSet.java:255)
at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
at java.util.TreeSet.addAll(TreeSet.java:312)
Voyence.log:
WARN : 2016-05-06 17:09:09,475: SchedulerServiceImpl: Job finished executing: jobId=10136
ERROR: 2016-05-06 17:15:12,985: DiskPageStore$SessionEntry: Error writing to a channel sun.nio.ch.FileChannelImpl@14da691
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731)
at org.apache.wicket.protocol.http.pagestore.DiskPageStore$SessionEntry.savePage(DiskPageStore.java:246)
at org.apache.wicket.protocol.http.pagestore.DiskPageStore.flushPagesToSaveList(DiskPageStore.java:895)
at org.apache.wicket.protocol.http.pagestore.DiskPageStore$PageSavingThread.run(DiskPageStore.java:965)
at java.lang.Thread.run(Thread.java:745)
Resolution
1. Clear out old logs in the /usr/tomcat/apache-tomcat-8.0.23/logs directory.
Generally, 30 days of logs is sufficient. Follow your company's retention policy for maintaining old log files.
2. Increase the max memory in .bash_profile (change to -Xmx=4096m) located in /usr/tomcat/apache-tomcat-8.0.23/
(This is a hidden file, use 'ls -al' to view the file.
After making the modification, restart services to re-read the .bash_profile).
Generally, 30 days of logs is sufficient. Follow your company's retention policy for maintaining old log files.
2. Increase the max memory in .bash_profile (change to -Xmx=4096m) located in /usr/tomcat/apache-tomcat-8.0.23/
(This is a hidden file, use 'ls -al' to view the file.
After making the modification, restart services to re-read the .bash_profile).
Additional Information
GC overhead limit exceeded refers to the Garbage Collector. When an object on the heap is no longer usable, it is marked for garbage collection. If/when your application is running low on memory the garbage collector will run, throwing out unreachable objects and freeing up space so it can be re-used. Garbage collection errors indicate that the JVM took too long to free up memory during its GC process.
The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small.
This kind of OutOfMemoryError can be caused if user requests drown the available resources in the JVM. When this occurs, performance will degrade aggressively.
Things to check:
1. Check for .hprof files in /var/cores or in /opt/smarts-ncm/dump/ . You can also run locate *.hprof to find and heap files. You will need to move these off of the server (or delete them) as they may be consuming space. The most recent dump file can be analyzed, if needed. Engineering will need the .hprof file and associated catalina.out and Voyence.log.
2. Check the file space for RA to be sure you are not out of space (No space left on device). Space is usually consumed by logs or hprof files. Logs are stored in /usr/tomcat/apache-tomcat-8.0.23/logs
3. Investigate RA activity. Are there multiple reports running sumultaneously? Are these reports large? (See the NCM support matrix documentation for more information)
4. Does the server have enough memory to support NCM Report Advisor? (See the NCM support matrix documentation for more information)
The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small.
This kind of OutOfMemoryError can be caused if user requests drown the available resources in the JVM. When this occurs, performance will degrade aggressively.
Things to check:
1. Check for .hprof files in /var/cores or in /opt/smarts-ncm/dump/ . You can also run locate *.hprof to find and heap files. You will need to move these off of the server (or delete them) as they may be consuming space. The most recent dump file can be analyzed, if needed. Engineering will need the .hprof file and associated catalina.out and Voyence.log.
2. Check the file space for RA to be sure you are not out of space (No space left on device). Space is usually consumed by logs or hprof files. Logs are stored in /usr/tomcat/apache-tomcat-8.0.23/logs
3. Investigate RA activity. Are there multiple reports running sumultaneously? Are these reports large? (See the NCM support matrix documentation for more information)
4. Does the server have enough memory to support NCM Report Advisor? (See the NCM support matrix documentation for more information)
Feedback
Yes
No
Powered by
