Patch failed with errors 'node has conditions: [DiskPressure]'

Products

• VMware Integrated OpenStack

Issue/Introduction

• viocli patch list reports that the patch is failed to install

patch-7.1.0.0-17987093 Failed

• Many pods are listed as evicted from controller nodes because of diskPressure:

kubelet.log
Jul 04 00:17:58 controller-r895wjp2r2 kubelet[470]: W0704 00:17:58.555104 470 eviction_manager.go:159] Failed to admit pod octavia-driver-agent-wwj2l_openstack(########-####-####-####-############) - node has conditions: [DiskPressure]

Jul 06 10:49:01 controller-dndgwnvswt kubelet[1394]: W0706 10:49:01.121724 1394 eviction_manager.go:159] Failed to admit pod fluentd-lrsd7_openstack(########-####-####-####-############) - node has conditions: [DiskPressure]

• The disk usage for controller node(s) is above 80%

[ ~ ]$ df -H
Filesystem Size Used Avail Use% Mounted on
devtmpfs 8.5G 0 8.5G 0% /dev
tmpfs 8.5G 0 8.5G 0% /dev/shm
tmpfs 8.5G 2.4M 8.5G 1% /run
tmpfs 8.5G 0 8.5G 0% /sys/fs/cgroup
/dev/sda4 27G 21G 4.3G 84% /         <<<<<<<
tmpfs 8.5G 0 8.5G 0% /tmp
/dev/sda2 126M 24M 96M 20% /boot
tmpfs 1.7G 0 1.7G 0% /run/user/1000

Environment

7.x

Cause

Not enough disk space left on the controllers(s)

Resolution

1. Check journal logs disk usage 

viocli get controllers
viossh <controller name>   (repeat for each controller)

journalctl --disk-usage 

root@controller-6n8wwfvqv8 [ / ]# journalctl --disk-usage
Archived and active journals take up 10.0G in the file system.

2. Vacuum the journal logs

journalctl --vacuum-size=1G
or
journalctl --vacuum-time=7days

Note:  This will shrink the journal.  Cleaning up older data in the journal logs.

3. Retry the patch operation after freeing up space 

If this issue persists, contact Broadcom Support and note this article in the problem description.