Cannot Create VKS Clusters - failed to cache storage items: failed to check if file exists: 500 Internal Server Error

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

VKS clusters will not create regardless of being created from VCFA or through Local Consumption Interface (LCI).

 

In the Supervisor cluster context, the following symptoms are observed:

• The VKS cluster object creates and the first control plane node has a machine, vspheremachine and virtualmachine created but does not progress from there:
  

  kubectl get cluster,machine,vspheremachine,vm -o wide -n <namespace>

   

• Describing the virtual machine for the control plane node of the VKS cluster shows the following error:
  

  kubectl describe vm -n <namespace> <vm name>
  
  message: 'failed to cache files: failed to cache storage items: failed to check if file exists: 500 Internal Server Error'

   

• The vmop-controller-manager manager container logs show the following errors for the VKS clusters that are failing to create where values in brackets <> will vary by environment:
  

  kubectl get pods -n vmware-system-vmop
  
  kubectl logs -n vmware-system-vmop -c manager <vmware-system-vmop-controller-manager-id>
  
  IMMDD HH:MM:SS.ssssss  1 recorder.go:104] "failed to cache files: failed to cache storage items: failed to check if file exists: 500 Internal Server Error" logger="events" type="Warning" object={"kind":"VirtualMachine","namespace":"<namespace>","name":"<vks cluster name>","uid":"<uid>","apiVersion":"vmoperator.vmware.com/v1alpha5","resourceVersion":"######"} reason="CreateFailure"
  
  EMMDD HH:MM:SS.ssssss       1 controller.go:474] "Reconciler error" err="failed to cache files: failed to cache storage items: failed to check if file exists: 500 Internal Server Error" logger="virtualmachine-controller" VirtualMachine="<namespace>/<vks cluster name>"

   

In the vCenter, vpxd logs report the following error across one or more host-ids:

YYYY-MM-DDTHH:MM:SS.sssZ INFO vpxd #### [vc@4413 sub="vmomi.soapStub[##]" opID="<opID>"] SOAP request returned HTTP failure; <, /hgw/<host-id>/vpxa>, method: retrieveVStorageObject; code: 500(Internal Server Error); fault: (vim.fault.NotFound) {

 

On the ESXi host where the VM is trying to provision, the following error is found in hostd logs:

WARNING Hostd <id> [esx@#### sub="Default" opID="<opID>" user="vpxuser:VSPHERE.LOCAL\wcp-vmop-user-<userid>"] Vslm Failure: VslmRetrieveDisk failed for fcd <FCD ID> on datastore /vmfs/volumes/<location>/ with type vim.fault.NotFound, N3Vim5Fault8NotFound9ExceptionE(Fault cause: vim.fault.NotFound

Environment

vSphere Supervisor

vSphere Cloud Foundation 9.X

Cause

A stale or orphaned FCD in the environment is preventing the creation of VMs for VKS cluster nodes.

Resolution

A reconciliation on the datastores in the environment will clean up any stale or orphaned FCDs, including the FCD causing this issue.

See Reconciling Discrepancies in the Managed Virtual Disk Catalog

NOTE: Depending on the size of the environment, datastore reconciliation task may show success but may not fully take effect for at least 2 hours.