Unable to create pvc/pv after configuring vsphere-csi on a Rancher K8.
search cancel

Unable to create pvc/pv after configuring vsphere-csi on a Rancher K8.

book

Article ID: 396184

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • On creating a test-pvc it goes into pending state.
  • vSphere-csi pod status will be healthy and running properly.

 

On describing the pending pvc, we observe following events : 

Name:          vsphere-pvc
Namespace:     name_space_name
StorageClass:  vsphere-csi-sc
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
               volume.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age                   From                                                                                                 Message
  ----     ------                ----                  ----                                                                                                 -------
  Normal   Provisioning          2m4s (x8 over 4m13s)  csi.vsphere.vmware.com_vsphere-csi-controller-568f4b5ff6-87gm2_d972e8c9-a4fd-43d4-a4a2-9ce1caa21d46  External provisioner is provisioning volume for claim "local/vsphere-pvc"
  Warning  ProvisioningFailed    2m3s (x8 over 4m13s)  csi.vsphere.vmware.com_vsphere-csi-controller-568f4b5ff6-87gm2_d972e8c9-a4fd-43d4-a4a2-9ce1caa21d46  failed to provision volume with StorageClass "vsphere-csi-sc": rpc error: code = Internal desc = failed to create volume. Error: auth service: no shared datastore found for block volume provisioning
  Normal   ExternalProvisioning  8s (x18 over 4m13s)   persistentvolume-controller                                                                          Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

 

From the vsphere-csi-controller logs :
"level":"info","time":"YYYY-MM-SSThh:mm:ss.333855488Z","caller":"common/authmanager.go:163","msg":"auth manager: datastoreMapForBlockVolumes is updated to map[] for vCenter \"lops-prif-vcsa1.autodatacorp.org\"","TraceId":"2c324b3a-c647-4d48-9466-128b9038d233"}
{"level":"warn","time":"YYYY-MM-SSThh:mm:ss.484670796Z","caller":"common/topology.go:456","msg":"failed to retrieve tags for category \"cns.vmware.topology-preferred-datastores\" in vCenter \"lops-prif-vcsa1.autodatacorp.org\". Reason: GET https://lops-prif-vcsa1.autodatacorp.org:443/rest/com/vmware/cis/tagging/category/id:cns.vmware.topology-preferred-datastores: 404 Not Found","TraceId":"d895608a-1cad-46b1-a2e4-4d952bf97c2b"}

From the vsphere-csi-provisioner logs :
I0501 hh:mm:ss.297213       1 event.go:364] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"local", Name:"vsphere-pvc", UID:"2c87f4c4-93c8-4298-a1f8-db35860ca8f7", APIVersion:"v1", ResourceVersion:"1429438", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "local/vsphere-pvc"
I0501 hh:mm:ss.649548       1 controller.go:1075] Final error received, removing PVC 2c87f4c4-93c8-4298-a1f8-db35860ca8f7 from claims in progress
W0501 hh:mm:ss.649576       1 controller.go:934] Retrying syncing claim "2c87f4c4-93c8-4298-a1f8-db35860ca8f7", failure 8
E0501 hh:mm:ss.649609       1 controller.go:957] error syncing claim "2c87f4c4-93c8-4298-a1f8-db35860ca8f7": failed to provision volume with StorageClass "vsphere-csi-sc": rpc error: code = Internal desc = failed to create volume. Error: auth service: no shared datastore found for block volume provisioning
I0501 hh:mm:ss.649641       1 event.go:364] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"local", Name:"vsphere-pvc", UID:"2c87f4c4-93c8-4298-a1f8-db35860ca8f7", APIVersion:"v1", ResourceVersion:"1429438", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "vsphere-csi-sc": rpc error: code = Internal desc = failed to create volume. Error: auth service: no shared datastore found for block volume provisioning

 

  • Verified the secret of vsphere-csi : k get secret -n vmware-system-csi -o yaml
    And, observed that the correct service account name ( SVC_Accpount_Name_Complete ) is given as updated in the rancher portal.
    Verified able to login to vCenter with the same service account.
  •  On checking the permissions of datastore (which will be used for this storage) with the same service account we see only a partial service account user name is listed i.e. "SVC_Accpount_Name_" instead of the complete  "SVC_Accpount_Name_Complete"

 

 

Environment

  • VMware vCenter Server 8.0
  • VMware vCenter Server 7.0

Cause

As vCenter is unable to read the complete service account name from the AD, it is capturing only upto a limited characters i.e. "SVC_Accpount_Name_" instead of the complete  "SVC_Accpount_Name_Complete"

Resolution

  • Update the service account name on the vsphere-csi driver from Rancher to the limited characters readable from vCenter i.e. "SVC_Accpount_Name_"
  • Then redeploy the vsphere-csi pods on the Rancher K8.
  • Then verify the that the secret is updated with the correct service account name:  k get secret -n vmware-system-csi -o yaml
  • Then re-create a test-pvc and it will successfully get recreated instantly.

Additional Information

Referenced document to configure vsphere-csi : Preparing for Installation of vSphere Container Storage Plug-in

Powered by Wolken Software