Platform VMs down with /tmp 100% full when patching from 6.14 to 6.14 P3 1738688587

Products

• VCF Operations for Networks

Issue/Introduction

• Use vLCM patch Aria Operations for Networks from 6.14 to 6.14 P3 1738688587.
• The patching failed and some services(Restapilayer/DatabusGateway/FlinkContainer) on platform VM are not running.
• Command 'df -h' returns /tmp/ directory usage 100%.
• /tmp/tools-bits/upgrade_patch_xxxxxxxxxx directories occupied most of the space in /tmp.

Resolution

1. Ssh to platform vm using support user.

2. Run following commands one by one:

sudo su
tdir="$(ls /tmp/tools-bits/ | wc -l)"
ls -lrt /tmp/tools-bits/ | head -n "$((tdir - 1))" | awk '{print $9}' | xargs -I{} rm -rf /tmp/tools-bits/{}
mkdir /tmp/infra-pkg

3. Wait for some time, the patch install should resume after this.