Cannot access login page on Security Analytics after a system crash

book

Article ID: 168596

calendar_today

Updated On:

Products

Security Analytics

Issue/Introduction

The redis server may not come up if the database is corrupted, which will prevent you from being able to log in to the GUI via a browser.

Cause

This could be caused by a power failure while the database is being updated by the application.

Possible symptoms are:
The GUI will not be available

From /var/log/messages:
Apr 30 12:54:46 vf_solera httpd: PHP Warning:  _cake_core_ cache was unable to write 'page_titles_eng' to Redis cache in /gui/lib/Cake/Cache/Cache.php on line 309
Apr 30 12:54:46 vf_solera httpd: PHP Warning:  _cake_core_ cache was unable to write 'controller_errors_eng' to Redis cache in /gui/lib/Cake/Cache/Cache.php on line 309
Apr 30 12:54:46 vf_solera httpd: PHP Warning:  _cake_core_ cache was unable to write 'file_map' to Redis cache in /gui/lib/Cake/Cache/Cache.php on line 309
Apr 30 12:54:46 vf_solera httpd: PHP Warning:  _cake_core_ cache was unable to write 'object_map' to Redis cache in /gui/lib/Cake/Cache/Cache.php on line 309
Apr 30 12:54:46 vf_solera httpd: PHP Warning:  _cake_core_ cache was unable to write 'method_cache' to Redis cache in /gui/lib/Cake/Cache/Cache.php on line 309


Other causes:
service redis restart does not start a redis-server process

ps -ef | grep redis

redis     4159     1  0 10:55 ?        00:00:08 /usr/sbin/redis-server /etc/redis.conf
root      9523  9477  0 12:29 pts/0    00:00:00 grep redis

tail /var/log/redis.log

[28710] 01 May 10:11:00 * Server started, Redis version 2.2.9
[28710] 01 May 10:11:00 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[28710] 01 May 10:11:00 # Short read or OOM loading DB. Unrecoverable error, aborting now.
[28784] 01 May 10:11:22 * Server started, Redis version 2.2.9
[28784] 01 May 10:11:22 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[28784] 01 May 10:11:22 # Short read or OOM loading DB. Unrecoverable error, aborting now.


Cannot start redis client
redis-cli
Could not connect to Redis at 127.0.0.1:6379: Connection refused


A redis dump shows:
redis-check-dump /var/lib/redis/dump.rdb

==== Processed 111 valid opcodes (in 255539 bytes) =============================
*** buffer overflow detected ***: redis-check-dump terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f076d9a7717]
/lib64/libc.so.6[0x7f076d9a55c0]
/lib64/libc.so.6[0x7f076d9a4949]
/lib64/libc.so.6(_IO_default_xsputn+0x96)[0x7f076d91e606]
/lib64/libc.so.6(_IO_vfprintf+0x348d)[0x7f076d8f0cbd]
/lib64/libc.so.6(__vsprintf_chk+0x9d)[0x7f076d9a49ed]
/lib64/libc.so.6(__sprintf_chk+0x80)[0x7f076d9a4930]
redis-check-dump[0x401f1c]
redis-check-dump[0x402113]
redis-check-dump[0x4023ca]
/lib64/libc.so.6(__libc_start_main+0xe6)[0x7f076d8c6546]
redis-check-dump[0x400a59]
======= Memory map: ========
00400000-00404000 r-xp 00000000 08:14 170767                             /usr/bin/redis-check-dump
00603000-00604000 rw-p 00003000 08:14 170767                             /usr/bin/redis-check-dump
00604000-0064b000 rw-p 00000000 00:00 0                                  [heap]
7f076d691000-7f076d6a7000 r-xp 00000000 08:14 160002                     /lib64/libgcc_s-4.3.2-20081105.so.1
7f076d6a7000-7f076d8a7000 ---p 00016000 08:14 160002                     /lib64/libgcc_s-4.3.2-20081105.so.1
7f076d8a7000-7f076d8a8000 rw-p 00016000 08:14 160002                     /lib64/libgcc_s-4.3.2-20081105.so.1
7f076d8a8000-7f076da10000 r-xp 00000000 08:14 160012                     /lib64/libc-2.9.so
7f076da10000-7f076dc10000 ---p 00168000 08:14 160012                     /lib64/libc-2.9.so
7f076dc10000-7f076dc14000 r--p 00168000 08:14 160012                     /lib64/libc-2.9.so
7f076dc14000-7f076dc15000 rw-p 0016c000 08:14 160012                     /lib64/libc-2.9.so
7f076dc15000-7f076dc1a000 rw-p 00000000 00:00 0
7f076dc1a000-7f076dc31000 r-xp 00000000 08:14 160036                     /lib64/libpthread-2.9.so
7f076dc31000-7f076de30000 ---p 00017000 08:14 160036                     /lib64/libpthread-2.9.so
7f076de30000-7f076de31000 r--p 00016000 08:14 160036                     /lib64/libpthread-2.9.so
7f076de31000-7f076de32000 rw-p 00017000 08:14 160036                     /lib64/libpthread-2.9.so
7f076de32000-7f076de36000 rw-p 00000000 00:00 0
7f076de36000-7f076deba000 r-xp 00000000 08:14 160020                     /lib64/libm-2.9.so
7f076deba000-7f076e0b9000 ---p 00084000 08:14 160020                     /lib64/libm-2.9.so
7f076e0b9000-7f076e0ba000 r--p 00083000 08:14 160020                     /lib64/libm-2.9.so
7f076e0ba000-7f076e0bb000 rw-p 00084000 08:14 160020                     /lib64/libm-2.9.so
7f076e0bb000-7f076e0db000 r-xp 00000000 08:14 160005                     /lib64/ld-2.9.so
7f076e28d000-7f076e2cc000 r--s 00000000 08:12 2613382                    /var/lib/redis/dump.rdb
7f076e2cc000-7f076e2ce000 rw-p 00000000 00:00 0
7f076e2d7000-7f076e2da000 rw-p 00000000 00:00 0
7f076e2da000-7f076e2db000 r--p 0001f000 08:14 160005                     /lib64/ld-2.9.so
7f076e2db000-7f076e2dc000 rw-p 00020000 08:14 160005                     /lib64/ld-2.9.so
7fff531cc000-7fff531ed000 rw-p 00000000 00:00 0                          [stack]
7fff531ff000-7fff53200000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

 

Resolution

To rebuild the redis database, run the following commands while logged in as root from the CLI:

1.   service redis stop
2.  rm -rf /var/lib/redis/dump.rdb
3.  rm -rf /var/run/redis.pid
4.  service redis start

Attempt to refresh the GUI login page to confirm that you can access it again.