API Gateway 10 Platform Patch is failing on some nodes

Products

• CA API Gateway

Issue/Introduction

The users are trying to apply the patch "Layer7_API_PlatformUpdate_64bit_v10.X-CentOS-2021-07-22.L7P" and the patch fails to install with these errors :-

Logs:

2021-08-21T03:36:22.466-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
2021-08-21T03:36:22.466-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: error: cannot open Packages index using db5 -  (-30973)
2021-08-21T03:36:22.466-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: error: cannot open Packages database in /var/lib/rpm
2021-08-21T03:36:22.466-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: warning: acl-2.2.51-15.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
2021-08-21T03:36:22.466-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: error: rpmdb: BDB0113 Thread/process 8689/139753370122304 failed: BDB1507 Thread died in Berkeley DB library
2021-08-21T03:36:22.466-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
2021-08-21T03:36:22.466-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: error: cannot open Packages database in /var/lib/rpm
2021-08-21T03:36:22.476-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: ERROR : Failed to install the missing RPMs
2021-08-21T03:36:22.481-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: Error installing patch: Error executing patch task: non-zero exit code returned from patch install script (returned 1)
2021-08-21T03:36:22.482-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: Full stack trace:
2021-08-21T03:36:22.482-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install: com.l7tech.server.processcontroller.patching.PatchException: Error executing patch task: non-zero exit code returned from patch install script (returned 1)
2021-08-21T03:36:22.483-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install:  at com.l7tech.server.processcontroller.patching.builder.ShellScriptPatchTask.runPatch(ShellScriptPatchTask.java:53)
2021-08-21T03:36:22.483-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Output from patch install:  at com.l7tech.server.processcontroller.patching.builder.PatchMain.main(PatchMain.java:37)
2021-08-21T03:36:22.646-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Patch exit code: 1
2021-08-21T03:36:22.647-0400 WARNING 258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Error installing Layer7_API_PlatformUpdate_64bit_v10.X-CentOS-2021-07-22
2021-08-21T03:36:22.647-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Patch files 'autoDelete' is ON; trying to delete patch ID: Layer7_API_PlatformUpdate_64bit_v10.X-CentOS-2021-07-22
2021-08-21T03:36:22.647-0400 INFO    258773 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Deleting patch ID: Layer7_API_PlatformUpdate_64bit_v10.X-CentOS-2021-07-22
2021-08-21T03:37:57.422-0400 INFO    258782 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Listing patches...
2021-08-21T03:39:22.623-0400 INFO    258784 com.l7tech.server.processcontroller.patching.PatchServiceApiImpl: Listing patches...

 

 

Environment

Release : 10.0

Component :

Cause

  The actual problem is caused by a corrupted rpm database.    

Resolution

The following solution has resolved the problem by rebuilding rpm database.

mkdir /var/lib/rpm/backup
cp -a /var/lib/rpm/__db* /var/lib/rpm/backup/

rm -f /var/lib/rpm/__db.[0-9][0-9]*
rpm --quiet -qa
rpm --rebuilddb