[VMC] Notification received "Workload VMs impacting upgrade activity"

Products

• VMware Cloud on AWS

Issue/Introduction

The organization has received a notification similar to below:

Please be advised that your SDDC upgrade is blocked because the VM <VM NAME> is not able to be migrated to other hosts. This VM is in an unhealthy state and you need power-off this VM to unblock the upgrade.

This will also block future planned maintenance activity scheduled on your SDDC (Like such as SDDC Upgrade, I3EN/PPG cluster conversion, ESXi Host maintenance activities).

The virtual machine has a failed migration task with a "Heap dvfilterVMotion" error:

Failed waiting for data. Error 195887124. Out of memory.

Environment

VMC on AWS

Cause

• The most common cause for this notification in VMC on AWS environments is due to dvfilter heap exhaustion. 
• When this occurs, the virtual machine is unable to migrate from the host. 
• This will block any current or upcoming maintenance and upgrades in the SDDC. 

Resolution

• To resolve this issue, power off the affected virtual machine. 
• This will free up the memory and allow the virtual machine to be migrated from the host. 
• This resolution is specific to VMC on AWS environments. 

Additional Information

For environments that are not VMC on AWS, follow this process to resolve dvfilter heap exhaustion: vMotion fails with Status - Failed to wait for data. Error 195887124. Out of memory