vMotion fails with error: "There are currently x usable fault domains. The operation requires x more usable fault domains. Failed to create object"

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• vMotion failure.
• vSAN environment
• Witness/data host going into not responding state intermittently
• vmkping between the data node or witness fails with 100 % packet loss.

 

Validation:

1. vMotion fails with error "There are currently 2 usable fault domains. The operation requires 1 more usable fault domains. Failed to create object"

Below messages are seen under vCenter UI > vSAN cluster > Host > Virtual machine>Tasks

 

2.Witness/data node going into not responding state intermittently.

Below messages are seen under vCenter UI > vSAN cluster > Host > Monitor >Events

 

 

3.Objects are in "reduced-availability-with-no-rebuild " on data node.

[root@Hostname :~] esxcli vsan debug object health summary get
Health Status                                              Number Of Objects
---------------------------------------------------------  -----------------
remoteAccessible                                                           0
inaccessible                                                               0
reduced-availability-with-no-rebuild                                       2
reduced-availability-with-no-rebuild-delay-timer                           0

4.vmkping between the data node and witness or other data node fails with 100 % packet loss.

[root@Hostname:] vmkping -I vmk0 ##.##.#.##
PING ##.##.#.## (##.##.#.##): 56 data bytes

--- ##.##.#.## ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss.

 

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

This issue is caused due to the vSAN cluster in a partitioned state, and the migration operation requires the vSAN to be in a healthy state. Witness host is partitioned state or isolated.

An insufficient number of Fault Domains are currently present in the environment for component placement. When Fault Domains are configured on the cluster, the number of Fault Domains takes precedence over the number of hosts present.

 

vSAN Witness node partitioned.

[root@Hostname:~] esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: YYYY-MM-DDTHH:MM:SSZ
   Local Node UUID: ########-####-####-####-############
   Local Node Type: WITNESS
   Local Node State: STANDALONE
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID:
   Sub-Cluster Backup UUID:
   Sub-Cluster UUID: ########-####-####-####-############
   Sub-Cluster Membership Entry Revision: 0
   Sub-Cluster Member Count: 1
   Sub-Cluster Member UUIDs: ########-####-####-####-############
   Sub-Cluster Member HostNames: ######
   Sub-Cluster Membership UUID: ########-####-####-####-############

Resolution

Ensure that all configured Fault Domains are online and all hosts are out of Maintenance Mode and are contributing to the vSAN cluster configuration.

To fix vSAN Witness partition in a stretched cluster issue  refer to: vSAN Witness appliance partitioned from stretch cluster

For more information on Network partitions in general please see: vSAN Health Service - Network Health - vSAN Cluster Partition

Once the partition issues are fixed the vMotion operation should be successful.

Additional Information

In a 3-node Cluster with 3 Fault Domains (1 Host per FD).

If One host has failed or partitioned and when you try to perform vMotion you may see below mesages  

"There are currently 2 usable fault domains. The operation requires 1 more usable fault domains."