Stopping all services through Ambari fails with "Error: rpmdb open failed"

Products

• Services Suite

Issue/Introduction

Symptoms:

When trying to stop all services through Ambari, the operation may fail with the following error messages:
 

Ambari will show the following error messages on the nodes marked in red:

2016-09-08 09:48:50,965 - Package['unzip'] {}
2016-09-08 09:48:51,003 - Installing package unzip ('/usr/bin/yum -d 0 -e 0 -y install unzip')
2016-09-08 09:48:51,110 - Error while executing command 'install':
Traceback (most recent call last):
 File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 123, in execute
 method(env)
 File "/var/lib/ambari-agent/cache/stacks/PHD/2.0.6/hooks/before-INSTALL/scripts/hook.py", line 34, in hook
 install_packages()
 File "/var/lib/ambari-agent/cache/stacks/PHD/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py", line 63, in install_packages
 Package(packages)
 File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 107, in __new__
 cls(name.pop(0), env, provider, **kwargs)
 File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
 self.env.run()
 File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
 self.run_action(resource, action)
 File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
 provider_action()
 File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 40, in action_install
 self.install_package(package_name)
 File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 36, in install_package
 shell.checked_call(cmd)
 File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 36, in checked_call
 return _call(command, logoutput, True, cwd, env, preexec_fn, user, wait_for_finish, timeout, path)
 File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in _call
 raise Fail(err_msg)
Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install unzip' returned 1. rpmdb: Thread/process 2335/139851783018240 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 - (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

Environment

Cause

The RPM database is corrupt and needs to be rebuilt. This is caused by a file system crash or file system corruption. /var/log/messages should be checked to look for hardware or operating system configuration errors.

Resolution

1. Log on, through SSH, to the node that is showing up in red in the Ambari shutdown operation (dn02 and dn05 above).

2. Confirm that the operating system or hardware is healthy by checking /var/log/messages and other system
monitoring tools.

3. Backup and remove the current RPM database files:

mv /var/lib/rpm/__db* /tmp/

4. Rebuild the RPM database:

rpm -rebuilddb

5. Clean the caches in yum and confirm that yum works correctly:

yum clean all