Persistent volume creation is failing on TKGI clusters using EBS CSI Driver on AWS

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)
• VMware Tanzu Kubernetes Grid Integrated Edition 1.x
• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

In TKGI environment where out-of-tree (EBS) CSI driver is installed following the doc below errors are observed on ebs-csi-node pod logs.

$ kubectl logs ebs-csi-node-xxx -n kube-system
Defaulted container "ebs-plugin" out of: ebs-plugin, node-driver-registrar, liveness-probe
I1203 09:02:27.936582       1 main.go:153] "Initializing metadata"
I1203 09:02:31.453210       1 metadata.go:48] "Retrieved metadata from IMDS"
I1203 09:02:31.454222       1 driver.go:69] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.37.0"
E1203 09:02:32.865556       1 node.go:857] "Unexpected failure when attempting to remove node taint(s)" err="isAllocatableSet: driver not found on node 
E1203 09:02:33.455947       1 node.go:857] "Unexpected failure when attempting to remove node taint(s)" err="isAllocatableSet: driver not found on node 
E1203 09:02:34.509197       1 node.go:857] "Unexpected failure when attempting to remove node taint(s)" err="isAllocatableSet: driver not found on node 
E1203 09:02:36.553760       1 node.go:857] "Unexpected failure when attempting to remove node taint(s)" err="isAllocatableSet: driver not found on node 

On reviewing the csi node configuration spec.drivers is null

$ kubectl get csinodes -A -o yaml
apiVersion: v1
items:
apiVersion: storage.k8s.io/v1
  kind: CSINode
  ...
  spec:
    drivers: null

Environment

TKGI on AWS

Cause

kubelet path in TKGi is /var/vcap/data/kubelet, but the manifest in these csi driver uses /var/lib/kubelet, so csi-driver fails to register to kubelet.

Resolution

Please add option "--set linux.kubelet=/var/vcap/data/kubelet" for helm install. 
Please replace "/var/lib/kubelet" with "/var/vcap/data/kubelet" in the manifest for manual install.

Additional Information

Same resolution applies for Azure environment when installing csi following below documents.
azuredisk-csi-driver
azurefile-csi-driver