One of our PAM servers had become unresponsive. Leading up to this we noticed rapid increase of disk utilization and suspected that the hard drive may be full.
Once Support was engaged, we were able to get to a point where the UI became accessible again, and indeed disk utilization was close to 100%.
Applet Log Level was set to Debug in Configuration > Diagnostics > Diagnostic Logs > Log Levels page for a long time, causing a log file related to session recording that normally does not grow and currently is not subject to log rotation to gradually fill up the disk.
Release : 3.4.x, 4.0.x
Component : PRIVILEGED ACCESS MANAGEMENT
When Applet Log Level is set to Debug, there is a known issue that causes verbose debug messages are written into /opt/tomcat-sessionRec/logs/catalina.out file. Please set back Applet Log Level to default value, i.e. Error, in Configuration > Diagnostics > Diagnostic Logs > Log Levels page to stop the catalina.out file to grow in size rapidly.
Please raise Technical Support ticket and engage PAM Support Engineer to SSH into the PAM node and empty the catalina.out file running the following command.
#echo "" > /opt/tomcat-sessionRec/logs/catalina.out
This is a known issue that is recorded in DE512177. At the time when this KB article is written, Broadcom Engineering investigating and planning to avoid this issue in future releases.
In almost all cases a reboot of the node will get a node with a full disk back to a state where the UI, either the regular UI or the config user page, becomes accessible again for a while, as the reboot will free up some disk space. This should allow Support to get SSH access and address the problem per instructions above. Hold off with the reboot until Support is engaged.