Deleted Virtual Machine caused some CNS tasks to be stuck in queued state.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

Operations sent to CNS returned a task id, but the task state never changes from queued. The user knows these tasks are for volumes attached to a virtual machine that is just deleted.

Environment

VMware vCenter Server 8.0.2

Cause

CNS recently introduced new serialization logic to schedule tasks conflict-free. This relies on virtual machine attachment information to be accurate. When the virtual machine is deleted before all its volumes operations can be scheduled, such attachment information is lost and confuses CNS scheduling.

 

 

Resolution

The issue is not fixable from CNS perspective. To avoid this issue, please wait for all the volumes operations to be completed before deleting the Virtual Machine.

Workaround:

To workaround the issue, please follow the below mentioned steps:

If application layer can fix the calling order, nothing needs to be done on CNS side. If not, CNS new serialization can be disabled in below steps:

1. Open /usr/lib/vmware-vsan/VsanVcMgmtConfig.xml on vCenter.

2. Change below config to false.

<newSerializationEnabled>true</newSerializationEnabled>

3. Restart vsan-health with 'vmon-cli -r vsan-health'.