Persistent volume creation is failing on TKGI clusters using EBS CSI Driver on AWS
search cancel

Persistent volume creation is failing on TKGI clusters using EBS CSI Driver on AWS

book

Article ID: 383792

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Integrated (TKGi) VMware Tanzu Kubernetes Grid Integrated Edition 1.x VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

In TKGI environment where out-of-tree (EBS) CSI driver is installed following the doc below errors are observed on ebs-csi-node pod logs.

$ kubectl logs ebs-csi-node-xxx -n kube-system
Defaulted container "ebs-plugin" out of: ebs-plugin, node-driver-registrar, liveness-probe
I1203 09:02:27.936582       1 main.go:153] "Initializing metadata"
I1203 09:02:31.453210       1 metadata.go:48] "Retrieved metadata from IMDS"
I1203 09:02:31.454222       1 driver.go:69] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.37.0"
E1203 09:02:32.865556       1 node.go:857] "Unexpected failure when attempting to remove node taint(s)" err="isAllocatableSet: driver not found on node 
E1203 09:02:33.455947       1 node.go:857] "Unexpected failure when attempting to remove node taint(s)" err="isAllocatableSet: driver not found on node 
E1203 09:02:34.509197       1 node.go:857] "Unexpected failure when attempting to remove node taint(s)" err="isAllocatableSet: driver not found on node 
E1203 09:02:36.553760       1 node.go:857] "Unexpected failure when attempting to remove node taint(s)" err="isAllocatableSet: driver not found on node 

On reviewing the csi node configuration spec.drivers is null

$ kubectl get csinodes -A -o yaml
apiVersion: v1
items:
apiVersion: storage.k8s.io/v1
  kind: CSINode
  ...
  spec:
    drivers: null

Environment

TKGI on AWS

Cause

kubelet path in TKGi is /var/vcap/data/kubelet, but the manifest in these csi driver uses /var/lib/kubelet, so csi-driver fails to register to kubelet.

Resolution

Please add option "--set linux.kubelet=/var/vcap/data/kubelet" for helm install. 
Please replace "/var/lib/kubelet" with "/var/vcap/data/kubelet" in the manifest for manual install.

Additional Information

Same resolution applies for Azure environment when installing csi following below documents.
azuredisk-csi-driver
azurefile-csi-driver