Persistent volume is failing to attach with Error: "The input volume <Volume handle ID> is not registered as a CNS volume"
search cancel

Persistent volume is failing to attach with Error: "The input volume <Volume handle ID> is not registered as a CNS volume"

book

Article ID: 320790

calendar_today

Updated On:

Products

VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

Issue: CSI volumeattachment is getting the below error while attaching Persistent Volume to a Kubernetes node.

Status:
  Attach Error:
    Message:  rpc error: code = Internal desc = failed to attach disk: "1e10bd8c-4713-4731-95df-*******" with node: "420c5615-7b83-a52f-90b8-*******" err failed to attach cns volume: "1e10bd8c-4713-4731-95df-********" to node vm: "VirtualMachine:vm-**** [VirtualCenterHost: *********, UUID: ********-7b83-a52f-90b8-********, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-****, VirtualCenterHost: *************]]". fault: "(*types.LocalizedMethodFault)(0xc000987360)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (types.CnsFault) {\n  BaseMethodFault: (types.BaseMethodFault) <nil>,\n  Reason: (string) (len=88) \"The input volume 1e10bd8c-4713-4731-95df-******** is not registered as a CNS volume.\"\n },\n LocalizedMessage: (string) (len=35) \"fault.CnsNotRegisteredFault.summary\"\n})\n". opId: "dde210ec"
    Time:     YYYY:MM:DD
  Attached:   false
  Detach Error:
    Message:  rpc error: code = Internal desc = volumeID "1e10bd8c-4713-4731-95df-********" not found in QueryVolume

Also, PV would not be listed on the vCenter UI under "Container Volumes". Refer the below screenshot

 

 

Environment

VMware Tanzu Kubernetes Grid
vSphere with Tanzu 7.0
vSphere with Tanzu 8.0

Cause

It can be caused due to general discrepancies between the datastore and the CNS database.

Resolution

Rebuild the FCD catalog and reconcile the datastore

NOTE: If the hosts are running vSAN, first run "esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates" on ALL hosts in the vSAN cluster. This will prevent vCenter from removing any hosts from the cluster when they stop communicating (due to stopping hostd). Once the procedure in this article is completed, re-enable vCenter member updates by running "esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates".

  1. Identify all affected datastores.
  2. Identify all ESXi hosts connected to the datastores for which the FCD catalog is being rebuilt.
  3. For each identified host, SSH into it and stop the hostd process.

        /etc/init.d/hostd stop

  4. Select one of the hosts that has the datastore mounted and SSH into it. Take a backup of the catalog folder for all the datastores listed above and restart hostd.

        cd /vmfs/volumes/<datastore>/
        mv catalog catalog_backup
        /etc/init.d/hostd start

                    NOTE: If running vSAN, files cannot be moved ("mv"), as it is object based storage. Contents will need to be moved, rather than the catalog directory itself as outlined below.

        cd catalog
        mv * /tmp/catalog

    5.  To use the ESXi MOB, the MOB needs to be enabled before accessing. Enable host MOB

              https://<ESXI_host_fqdn/IP>/mob/?moid=ha-vstorage-object-manager&method=reconcileDatastoreInventory       

              Path: MOB > content > vStorageObjectManager > HostReconcileDatastoreInventory_Task

         If running vSAN, the reconcile task is to be run on the vCenter MOB rather than the ESXI host.

                 https://<vcsa- fqdn>/mob/?moid=VStorageObjectManager&method=reconcileDatastoreInventory

             NOTE: Stopping of hostd process should completed for all hosts connected to the datastore

     6. Enter the datastore URL that need to be reconciled. 
             Example: ds:///vmfs/volumes/<datastore-UUID>/ (Found on the summary page of the datastore in the vCenter UI)
             If using vCenter MOB, use the datastore MOID, e.g., "datastore-##" (From the URL when the datastore is selected in the vSphere client)

    7.   Select  invoke to complete reconcile task for affected datastore. Wait until the task state shows as "Success".

    8. SSH into the remaining hosts and start the hostd process.

      /etc/init.d/hostd start

    9. If the database content is not getting updated even after some time, and the database is showing old content, it may require triggering a sync for "StorageLifecycleManager".

       https://<VCIP>/vslm/mob//?moid=StorageLifecycleManager&method=VslmSyncDatastore

   10. The datastore URL will be like "ds:///vmfs/volumes/##### - ###### - ######". (Found on the summary page of the datastore in the vCenter UI)

        Set "fullSync=true"
    "fcd Id" can be blank


   11. Once this is done, wait for some time for data sync between sps and CNS. After this sync, the Persistent volume will show up in vCenter's CNS UI.

  

Additional Information

Powered by Wolken Software