[Internal] Persistent volume for console logs is 100% full in VIO

Products

• VMware Integrated OpenStack

Issue/Introduction

Symptoms:
In the nova-vspc/0.log in VMware Integrated OpenStack you will see the following error message:

XXXX-XX-XXTXX:XX:XXZ future: <Task finished coro=<VspcServer.handle_telnet() done, defined at /usr/lib/python3.7/site-packages/vspc/server.py:198> exception=OSError(28, 'No space left on device')>: OSError: [Errno 28] No space left on device: '/var/log/vspc/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
XXXX-XX-XXTXX:XX:XXZ XXXX-XX-XX XX:XX:XX.XXX 1 ERROR asyncio Traceback (most recent call last):
XXXX-XX-XXTXX:XX:XXZ XXXX-XX-XX XX:XX:XX.XXX 1 ERROR asyncio   File "/usr/lib/python3.7/site-packages/vspc/server.py", line 213, in handle_telnet
XXXX-XX-XXTXX:XX:XXZ XXXX-XX-XX XX:XX:XX.XXX 1 ERROR asyncio     self.save_to_log(uuid, data)
XXXX-XX-XXTXX:XX:XXZ XXXX-XX-XX XX:XX:XX.XXX 1 ERROR asyncio   File "/usr/lib/python3.7/site-packages/vspc/server.py", line 195, in save_to_log
XXXX-XX-XXTXX:XX:XXZ XXXX-XX-XX XX:XX:XX.XXX 1 ERROR asyncio     with open(fpath, 'ab') as f:
XXXX-XX-XXTXX:XX:XXZ XXXX-XX-XX XX:XX:XX.XXX 1 ERROR asyncio OSError: [Errno 28] No space left on device: '/var/log/vspc/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
XXXX-XX-XXTXX:XX:XXZ XXXX-XX-XX XX:XX:XX.XXX 1 ERROR asyncio ^[[00m
XXXX-XX-XXTXX:XX:XXZ XXXX-XX-XX XX:XX:XX.XXX 1 INFO vspc.server [-] ('XXX.xx.xx.x', XXXXX) connected^[[00m

Environment

VMware Integrated Openstack 7.x

Cause

The persistent volume for console logs has a max size of 2GB by default. There is no rotation of logs or deletion with instances.

Resolution

We can increase the size of the persistent volume by following these steps:

1. SSH as root to the VIO Management Server.
2. Obtain the list of nova-compute by running this command: viocli get novacompute
3. Update nova-compute with: viocli update novacompute <nova-compute UUID from above>
4. Add the following at the end of the file in this exact format (with desired size being 4 GB as an example):

storage:
  pvc:
    size: 4Gi

5. Save and exit the file with :w and :q.
6. Retrieve the compute statefulset with: osctl get statefulset |grep compute
7. Open the statefulset for editing with: osctl edit statefulset <compute statefulset from the above output>
8. Change the replicas from 1 to 0. It should look like the following:

spec:
  podManagementPolicy: OrderedReady
  replicas: 0
  revisionHistoryLimit: 10

9. Save and exit the file the same as step 5.
10. Retrieve the list of persistent volume containers with: osctl get pvc
11. Find the vspc container and delete it with: osctl del <vspc container from the above list>
12. After the container deletes run the following command to watch for compute re-building pods with: pods
13. Once compute has initialized verify that the container is the desired size with: osctl get pvc
14. Repeat as necessary if there are any failures.

Workaround:
As a workaround we can delete log files on the persistent volume container that are taking up too much space.

1. SSH to the VIO Management Server.
2. Retrieve the list of compute pods with: osctl get pods|grep compute
3. Tunnel into the compute pod with: kubectl exec -it <pod> -n openstack /bin/bash
4. Change into the volume's mounted directory: cd /var/log/vspc
5. Run this command to have the log files listed from smallest to largest: ls -lSr
6. Delete as the customer sees fit.

Additional Information

Impact/Risks:
Before following the steps given make sure the customer has recent backups of their VMware Integrated OpenStack deployment. The resolution and workaround will both delete current console logs.

Note  verify that the customer agrees to the deletion of the current console logs held on the container.