Error Applying Gateway Patch

Products

• CA API Gateway

Issue/Introduction

Attempting to install a monthly platform patch to our gateway resulted in the following output:

You have chosen to install the following patch

" Layer7_API_PlatformUpdate_64bit_v10.X-CentOS-2023-03-23 "

Press [Enter] to continue

Please wait while the patch is installed ...

Patch ID Layer7_API_PlatformUpdate_64bit_v10.X-CentOS-2023-03-23 (Upgrades the Layer7 API 64bit Platform to the 10.X-CentOS-2023-03-23 version. This patch requires that the 10.X Platform Update be installed first. It also requires a restart of the Layer7 API Gateway Appliance.) is ERROR, last modified on xxxxx.

There were errors during the patch operation. See sspc logs in /opt/SecureSpan/Controller/var/logs for details.

Environment

CA API Gateway 10.x , 11.0

Cause

In this particular situation we were able to determine that all of the inodes were exhausted on the /var partition on the gateway by executing the following command:

df -ki

Filesystem                           Inodes  IUsed    IFree  IUse%    Mounted on
/dev/mapper/vg00-lv_var    196608 196608       0   100%     /var

 

 

Resolution

With this information we suspected that there had to probably be thousands of small files on the system which needed to get cleaned up and removed.

We looked to determine where these files were located by executing the following command directly on that problematic node:

for i in /var/*; do echo $i; find $i |wc -l; done

That will list out the total number of inodes in each directory, find the directory with a large number of inodes and continue to execute a similar kind of command but specify the directory in question with the large number of occupied inodes.  If the above command shows thousands of inodes in use within /var/lib then we can further narrow in on where exactly those files were located by executing:

for i in /var/lib/*; do echo $i; find $i |wc -l; done

 

In this particular case we found the /var/spool/clientmqueue had an excessive number of files within it.  Once those files were cleared the gateway patch installed with no issues.