Error: "failed to create object: No space left on device" when deploying a VIC container in vSAN

Products

• VMware vSphere ESXi

Issue/Introduction

• Deploying a VIC container on vSAN fails due to available free space in vSAN.
• Checking the storage usage shows there is available free space.
• In the hostd.log file, you see error similar to:
  
  yyy-mm-ddTHH:MM:SS.###Z info hostd[43D40B70] [Originator@6876 sub=Libs opID=795322a2-5-6b67 user=vpxuser:VSPHERE.LOCAL\Administrator] OBJLIB-VSANOBJ: VsanObjCreate: failed to create object: No space left on device (1835010).
  
  
• In the vpxd.log file in vCenter Server, you see similar error after attempting to run a container:
  
  yyy-mm-ddTHH:MM:SS.###Z info vpxd[7F96047BC700] [Originator@6876 sub=Default opID=795322a2] [VpxLRO] -- ERROR task-165691 -- vm-16302 -- vim.VirtualMachine.reconfigure: vmodl.fault.SystemError:
  --> Result:
  --> (vmodl.fault.SystemError) {
  --> faultCause = (vmodl.MethodFault) null,
  --> reason = "Out of resources",
  --> msg = "A general system error occurred: Out of resources"
  --> }
  ####Logs Omitted For Clarity####
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
  
  For more information, see: Location of log files for VMware products (1021806)

Environment

VMware vSphere Integrated Containers 1.0.x

Cause

This issue occurs when the backing vSAN object is not removed in the delete workflow after deleting the VIC Container. When vSAN objects are still present, you cannot create a new object as the maximum object capacity is already met.

Resolution

Prerequisites:

Before proceeding, confirm the deployment is impacted by this issue by the following steps.

1. Verify that the deployment is on vSphere Integrated Containers v0.8.0-7315-c8ac999 (Server Version), by running this command from a docker client:
   
   docker -H ###.###.#.###:#### --tls info
   
   
2. Check if the ESXi host detects free space on the vSAN datastore by running this command:
   
   Note: Connect to the shell of the ESXi host that attempted to run the container.
   
   df -f
   
   Verify the datastore had enough available space for the container's.
   
   
3. Check if the Distributed Object Manager (DOM) is at maximum capacity by running this command:
   
   cmmds-tool find -f json -t NODE_USAGE
   
   Compare the maxComponents to the componentsReserved.
   
   For example:
   
   {
   "uuid": "#############",
   "owner": "#############",
   "health": "Healthy",
   "revision": "88",
   "type": "NODE_USAGE",
   "flag": "2",
   "minHostVersion": "0",
   "md5sum": "############",
   "valueLen": "40",
   "content": "{\"maxComponents\": 9000, \"dataComponents\": 5888, \"componentsReserved\": 9000, \"leafNum\": 1542, \"leafLimit\": 36000}",
   "errorStr": "(null)"
   }

This is a known issue affecting vSphere Integrated Containers. Currently, there is no resolution.

Workaround:

Clean the vSAN objects. To clean vSAN obejects, open a case with Broadcom Support and quote this Knowledge Base article ID (340340) in the problem description. 

Additional Information

Location of log files for VMware products