Error: "Cannot complete file creation operation. There are currently 3 usable fault domains. The operation requires 1 more usable fault domain. Failed to create object." in the vCenter UI

Products

• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

Symptoms:

• Creating a virtual machine or changing the storage policy may fail with the below error in the vCenter UI:

"Cannot complete file creation operation. There are currently 3 usable fault domains. The operation requires 1 more usable fault domain. Failed to create object." 

Validation:

• /var/run/log/clomd.log will log the below event:

"CLOM_Diagnose: There are currently 3 usable fault domains. The operation requires 1 more usable fault domain."

• Verify the component limit and utilization using the below commands on vSAN ESXi host.

   # esxcli vsan debug limit get

   Component Limit Health: Red
   Disk Free Space Health: Green
   Free Components: 0 ----------------------> No free components on host <----
   Lowest Free Disk Space: 66 GB
   Max Components: 9000 -----------------> Max Component Limit <--------------
   Read Cache Free Reservation Health: Green
   Reserved Read Cache Size: 0
   Reserved Read Cache Size (GB): 0.00 GB
   Total Disk Space: 54,563,109,826,560 bytes
   Total Disk Space (GB): 50,815.86 GB
   Total Read Cache Size: 0
   Total Read Cache Size (GB): 0.00 GB
   Used Disk Space: 16,993,805,907,540 bytes
   Used Disk Space (GB): 15,826.72 GB

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

When vSAN host reaches the Max Component Limit , the object creation is expected to fail since component placement would not be allowed and fail.

Resolution

• Verify if the components are consumed by actual data written on the vSAN host. If the component utilization is matching with actual data, we may have to add more hosts to vSAN. Refer: vSAN/Skyline Health - Limits - Host component

• If the component count is not matching with the actual data, you may review unassociated objects which may be consuming the components on hosts. Refer: Procedures for identifying Unassociated vSAN objects

• You may reduce the component consumption on hosts by changing the storage policy for the virtual machines. Please refer: Change Storage Policy Assignment for Virtual Machine Files and Disks

  Notice: The change of storage policy may need more components and additional space on the hosts for the operation to complete. Please assess the storage space and available component limit on hosts before changing the policy.

 

Reducing the components utilization on host experiencing the Max Limit would allow the component creation on hosts to allow the object creation to work as expected.

Also, adding more hosts with disk group would allow the object creation to succeed  on vSAN cluster when other hosts have reached maximum.

Additional Information

vSAN Configuration Limits

Component utilization :-
 
•    Red: utilization > 90% for any host in this cluster;
•    Yellow: utilization <= 90% while utilization > 80% for any host in this cluster;
•    Green: utilization <= 80% for any host in this cluster;