VIO Management server tmp directory runs out of space during upgrade

Products

• VMware
• VMware Integrated OpenStack

Issue/Introduction

Symptoms:

• Happens on VIO 5.x upgrades
• /tmp/pre_upgrade directory of OMS VM is mounted by database01 VM via NFS during upgrade but it is getting full.
• You might see errors like this:

FAILED! => {"changed": true, "cmd": ["innobackupex", "--user=root", "--password=bsiF2c0d237Aeb56EF0c", "/mnt/backup/vio_os_db_20210410120315", "--no-timestamp", "--galera-info"], "delta": "0:01:55.308556", "end": "2021-04-10 12:05:17.403775", "failed": true

• tail -f /var/log/syslog

rpc.mountd[22513]: qword_eol: fprintf failed: errno 2 (No such file or directory)
Apr 10 12:57:27 rpc.mountd[22513]: qword_eol: fprintf failed: errno 2 (No such file or directory)

Environment

VMware Integrated OpenStack 6.x
VMware Integrated OpenStack 5.x
VMware Integrated Openstack 7.x

Resolution

1. Add extra 30GB disk to OMS VM in vCenter.
2. Rescan the devices on VIO5.1 OMS server.

echo "- - -" > /sys/class/scsi_host/host2/scan

 

3. Create partition on the new disk.

Find the new disk:
lsblk

Format disk
fdisk /dev/sdc1 

 

4. Create new Volume Group and a Logical Volume from the new disk

Create new Volume Group and a Logical Volume from the new disk
vgcreate vg02 /dev/sdc1
lvcreate -L 27G -n upgrade vg02

Create ext4 filesystem
mkfs.ext4 /dev/mapper/vg02-upgrade

Find UUID of /dev/mapper/vg02-upgrade and edit fstab
blkid
vi /etc/fstab
UUID=6d1d058a-034c-43b6-b35b-7af124b8936f /tmp  ext4 defaults    0 2

 

5. Mount /tmp directory on a new volume.

Mount /tmp directory
mount /tmp