DRS cannot find a host to power on virtual machine while deploying MinIO Tenants in vSphere Kubernetes cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• MinIO Tenant deployments fail to complete in a specific namespace, while deployments in other namespaces function correctly.

• MinIO pods in the affected namespace remain Pending indefinitely.

• Describing the pending pod (kubectl describe pod) reveals the following Kubernetes scheduler warning:
  Warning FailedScheduling default-scheduler 0/7 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling.

• Simultaneously, the vSphere Client reports a DRS failure for the underlying Virtual Machines:
  DRS cannot find a host to power on or migrate the virtual machine.

• The /var/log/vmware/vpxd.log on vCenter Server indicates that Virtual Machine creation was rejected due to insufficient memory or CPU capacity.

  warning vpxd[06815] [Originator@6876 sub=cdrsPlmt opID=####-####-####-#] Xdrs: no valid host for VM [vim.VirtualMachine:FakeVm-########,<unknown>] cluster [vim.ClusterComputeResource:domain-#####,#####] datastore Datastorename
info vpxd[06815] [Originator@6876 sub=drmLogger opID=####-####-####-#] Number of placement results(disks) added: 1.
info vpxd[06815] [Originator@6876 sub=drmLogger opID=####-####-####-#] PlaceDisk dump section ###### created.
error vpxd[06815] [Originator@6876 sub=drmLogger opID=####-####-####-#] Can not place disk.
warning vpxd[06815] [Originator@6876 sub=cdrsPlmt opID=####-####-####-#] Xdrs: no valid datastore for VM [vim.VirtualMachine:FakeVm-########,<unknown>] in cluster [vim.ClusterComputeResource:domain-#####,#####]
warning vpxd[06815] [Originator@6876 sub=cdrsPlmt opID=####-####-####-#] Xdrs: no valid recommendation for VM [vim.VirtualMachine:FakeVm-########,<unknown>] in cluster [vim.ClusterComputeResource:domain-#####,#####]
warning vpxd[06815] [Originator@6876 sub=MoDatacenter opID=####-####-####-#] Placing multi VMs failed with reason: xvmotionPlacement

  info vpxd[06815] [Originator@6876 sub=cdrsPlmt opID=####-####-####-#] Vm [vim.VirtualMachine:FakeVm-########,<unknown>] failed constraint check true on host [vim.HostSystem:host-####,hostname] with <objxmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:####" versionId="8.0.3.0" xsi:type="LocalizedMethodFault"><faultxsi:type="InsufficientHostMemoryCapacityFault"><faultMessage><key>com.vmware.drs.constraints.mem.InsufficientHostMemoryCapacityFault</key><arg><key>requested</key><value xsi:type="xsd:long">372961705984</value></arg><arg>
<key>unreserved</key><value xsi:type="xsd:long">298009493504</value></arg><arg><key>host</key><value xsi:type="xsd:string">hostname</value></arg></faultMessage><host type="HostSystem">host-####</host><unreserved>298009493504</unreserved><requested>372961705984</requested></fault><localizedMessage></localizedMessage></obj>

Environment

• VMware vSphere Kubernetes Service
• vCenter Server 8.x

Cause

• This issue occurs due to Insufficient Host Memory or CPU Capacity on the underlying ESXi cluster.
• The combined resource reservation requests from active workloads (such as MinIO Tenants in other namespaces) and the new deployment exceed the total available physical resources of the cluster.

Resolution

"Right-size" the Tenant configuration to fit within the available physical cluster capacity.

Step1: Modify the MinIO Tenant Configuration
             Reduce the resource requests in the MinIO Tenant specification to ensure compatibility with the remaining cluster capacity.

Step2: Redeploy
             Deploy a new tenant after modifying the resource allocation (lower CPU and Memory requests) to match the available capacity. Ensure that the old pods/tenants are removed before the new tenants are deployed

Additional Information

DRS Troubleshooting Information
MinIO Operator