vCenter UI reports "The object or item referred to could not be found from com.vmware.cns" task in Kubernetes environment

Products

• VMware vCenter Server

Issue/Introduction

In vSphere environments integrated with Kubernetes (including VKS, Tanzu, OpenShift, or upstream K8s) using the vSphere Container Storage Interface (CSI), the vCenter Server UI Task console displays continuous failures. These tasks are triggered automatically by the Cloud Native Storage (CNS) health check or volume reconciliation loops and fail immediately.

• vCenter UI Tasks console shows repeated failures from the com.vmware.cns component.
  
  
• Error Message: The object or item referred to could not be found
  
  
• In the vCenter /var/log/vmware/vpxd/vpxd.log, entries similar to the following are observed:
  
  

  [YYYY-MM-DDTHH:MM:SS] error vpxd[1017606] [Originator@6876 sub=Default opID=wcp-vCLS-93] [VpxLRO] -- ERROR lro-743409082 -- 520###db-3363-54f1-224b-ae1b####0491(52b##4fa-9fc4-eef6-0042-169d###216e8) -- propertyCollector -- vmodl.query.PropertyCollector.retrieveContents: :vmodl.fault.ManagedObjectNotFound
  --> Result:
  --> (vmodl.fault.ManagedObjectNotFound) {
  --> faultCause = (vmodl.MethodFault) null,
  --> faultMessage = <unset>,
  --> msg = "Received SOAP response fault from [ << io_obj p:0x00007f86c0114cc8, h:70, <UNIX ' '>, <UNIX ' /var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-6153/vpxa>] : retrieevstorageObject
  --> Received SOAP response fault from [ << io_obj p:0x000000947f68ad30, h:25, <TCP '127.0.0.1 : 58998'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: retrieveVStorageObject
  --> The object or item referred to could not be found.
  --> Arg datastore:
  --> "vim. Datastore:datastore-####'
  --> }

Environment

vCenter Server 7.x

vCenter Server 8.x

Cloud Native Storage

 

Cause

The issue is caused by a Cross vCenter Storage Configuration. The Kubernetes nodes are utilizing a shared datastore that is physically or logically presented to multiple, separate vCenter Server instances.

vSphere Container Storage Plug-in does not support datastores shared across vCenter instances. The cross vCenter storage presentation dependency is an architectural limitation and expected behavior

Resolution

To resolve this, the storage architecture must be aligned with the supported 1:1 mapping between a Kubernetes cluster and a single vCenter Server management domain.

Kindly reach out to Kubernetes vendor for making configuration changes following the design limitation as per the Broadcom document.

Additional Information

Refer to the vSphere Cloud Native Storage documentation for the specific CSI driver version being used to verify the "Limits and Constraints" section regarding Multi-vCenter support.

Compatibility Matrices for vSphere Container Storage Plug-in

vSphere Functionality Supported by vSphere Container Storage Plug-in