VIO vMotion of an instance fails with the error: Failed waiting for data

Products

• VMware
• VMware vSphere ESXi
• VMware Integrated OpenStack

Issue/Introduction

Symptoms:

• VIO vMotion of an instance fails with the error: 

Failed waiting for data.

• In vmware.log on the destination ESXi host, you can see logs similar to

2021-08-02T07:05:23.279Z| vmx| I125: [msg.migrate.waitdata.platform] Failed waiting for data.  Error bad003f. Connection closed by remote host, possibly due to timeout.

• In vmware.log on the source ESXi host, you can see logs siilar to

2021-08-02T07:05:23.221Z| vcpu-0| I125: serial0: Connection to vSPC was not established in specified time limit.
2021-08-02T07:05:23.221Z| vcpu-0| I125: CPT: error saving group serial0, 0

Environment

VMware Integrated Openstack 7.x
VMware Integrated OpenStack 6.x

Cause

For each instance, serial console logs are sent to VIO controllers via vspc connections.
If /var/log/vspc on these controllers is full of instance console logs, this vspc connection cannot write logs to this device due to "No space left on device".  As a result, vMotion for that instance will fail.

Resolution

This is a known issue affecting VMware Integrated Openstack 7.x

Workaround:
As a workaround, we can delete log files on the persistent volume container that are taking up too much space.

1. SSH to the VIO Management Server.
2. Retrieve the list of compute pods with: 

osctl get pods|grep compute

3. Tunnel into the compute pod with: 

kubectl exec -it <pod> -n openstack /bin/bash

4. Change into the volume's mounted directory: 

cd /var/log/vspc

5. Run this command to have the log files listed from smallest to largest: 

ls -lSr

6. Delete files as the customer sees fit.