Issue/Introduction:
The application monitor for Check Point Security Gateway R75 may fail if the broadcast address of the circuit for the synchronization network matches the pattern *.255.255.255.There are two symptoms for this issue:
1. The application status is Down (show application vap-group) and traffic is dropped by the NPM. The show flow active command reports action Drop (Load-balance failed) for the impacted traffic.
2. The disk usage on the /tftpboot partition is quickly growing. The disk usage can be verified with show disk-usage.
Cause:
A memory allocation error was found in the app_status binary that is causing this issue. It has been observed in some configurations when the broadcast address is set to *.255.255.255 on the synchronization circuit.
When running the program app_status manually, it produces following error output:
CBS# unix su
[[email protected] admin]# rsh fw_1
fw_1 (CBS): ~# /crossbeam/apps/app_status -v
*** glibc detected *** ./app_status: double free or corruption (out):
0x086d8c40 *** ======= Backtrace: ========= /lib/libc.so.6[0x56d666]
/lib/libc.so.6(cfree+0x59)[0x56daa9]
/opt/CPshrd-R75/lib/libOS.so(fw_free+0xeb)[0x18f51b]
/opt/CPshrd-R75/lib/libDataStruct.so(hash_delete+0x5e)[0x25a5de]
/opt/CPshrd-R75/lib/libDataStruct.so(_ZN6FwAtom14final_destructEv+0x41)[0x2690a1]
... etc.
Each time the program fails it writes a core file in the root directory on the VAP. Since the binary is executed every 5 seconds on each VAP with a Check Point application, the file system may become full quickly.
fw_1 (CBS): ~# cd /
fw_1 (CBS): /# ls -l core.app_status.*
bash: /bin/ls: Argument list too long
fw_1 (CBS): /# find . -name 'core*' | xargs ls -l | head
-rw------- 1 root root 1613824 Nov 1 06:55 ./core.app_status.1320144942.15744
-rw------- 1 root root 1613824 Nov 1 06:55 ./core.app_status.1320144947.15747
-rw------- 1 root root 1613824 Nov 1 06:56 ./core.app_status.1320144977.16534
-rw------- 1 root root 1613824 Nov 1 06:56 ./core.app_status.1320144997.16546
-rw------- 1 root root 1613824 Nov 1 06:56 ./core.app_status.1320145012.16553
-rw------- 1 root root 1613824 Nov 1 06:56 ./core.app_status.1320145017.16556
-rw------- 1 root root 1613824 Nov 1 06:57 ./core.app_status.1320145038.17341
-rw------- 1 root root 1613824 Nov 1 06:57 ./core.app_status.1320145048.17347
-rw------- 1 root root 1613824 Nov 1 06:57 ./core.app_status.1320145053.17391
-rw------- 1 root root 1613824 Nov 1 06:57 ./core.app_status.1320145063.17397
Resolution:
Contact Crossbeam support to receive a patched app_status binary.
To delete the core files and free up the disk, you can utilize the command find on the CPM unix command line:
CBS# unix su
[[email protected] admin]# find /tftpboot -maxdepth 2 -name 'core.app_status.*' | xargs rm -v
Workaround
There are two available workarounds:
1. Change the IP addressing schema for the sync network so that the broadcast address doesn't match the above pattern.
2. Alternatively disable application monitoring for the VAP group running Check Point SG R75, e.g.:
CBS# configure vap-group fw no application monitor
CBS# unix su
[[email protected] admin]# chmod -x /tftpboot/fw_common/crossbeam/apps/app_statusThe first command allows the NPM to load balance traffic even if application status is Down. The other command removes the executable bit of the binary so that it doesn't run. It is required to stop filling up the disk with new core files. The bit needs to be restored when a corrected
app_status is installed (
chmod +x).