Error: "exec: "lsblk": executable file not found in $PATH" causes pod to stay in ContainerCreating state in TKG Cluster

search cancel

Error: "exec: "lsblk": executable file not found in $PATH" causes pod to stay in ContainerCreating state in TKG Cluster

book

Article ID: 440052

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

Pods running in a TKG Cluster deployed using VMware Cloud Director Container Service Extension (CSE) stay in a ContainerCreating state.
The Kubernetes Container Storage Interface Driver for VMware Cloud Director (CSI) driver is not able to mount the persistent volumes into the container.
The Named Disk backing the persistent volume is created successfully in Cloud Director and attached to the Worker Node VM.
Describing the pod using kubectl describe commands shows FailedMount Events:kubectl describe pod -n <namespace_name> <pod_name>

Warning FailedMount #s (x# over #m) kubelet MountVolume.MountDevice failed for volume "pvc-########-####-####-####-############" : rpc error: code = Internal desc = unable to format and mount device [/dev/sdb] at path [/var/lib/kubelet/plugins/kubernetes.io/csi/named-disk.csi.cloud-director.vmware.com/################################################################/globalmount] with fs [ext4] and flags [[rw]]: [exec: "lsblk": executable file not found in $PATH]
Viewing the logs for the vcd-csi-plugin containers on csi-vcd-nodeplugin pods on the Worker Nodes show it is unable to format and mount the device:

kubectl logs -n kube-system csi-vcd-nodeplugin-##### -c vcd-csi-plugin

<time_stamp> 1 driver.go:215] GRPC error: function [/csi.v1.Node/NodeStageVolume] req [&csi.NodeStageVolumeRequest{VolumeId:"pvc-########-####-####-####-############", PublishContext:map[string]string{"diskID":"pvc-########-####-####-####-############", "diskUUID":"########-####-####-####-############", "filesystem":"ext4", "vmID":"<worker_node_vm_name>"}, StagingTargetPath:"/var/lib/kubelet/plugins/kubernetes.io/csi/named-disk.csi.cloud-director.vmware.com/################################################################/globalmount", VolumeCapability:(*csi.VolumeCapability)(0xc000422f40), Secrets:map[string]string(nil), VolumeContext:map[string]string{"busSubType":"VirtualSCSI", "busType":"SCSI", "diskID":"urn:vcloud:disk:########-####-####-####-############", "filesystem":"ext4", "storage.kubernetes.io/csiProvisionerIdentity":"#############-####-named-disk.csi.cloud-director.vmware.com", "storageProfile":"<storage_policy_name>"}, XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}]: [rpc error: code = Internal desc = unable to format and mount device [/dev/sdb] at path [/var/lib/kubelet/plugins/kubernetes.io/csi/named-disk.csi.cloud-director.vmware.com/################################################################/globalmount] with fs [ext4] and flags [[rw]]: [exec: "lsblk": executable file not found in $PATH]]

Environment

VMware Cloud Director 10.6.x
VMware Cloud Director Container Service Extension 4.2.4
Kubernetes Container Storage Interface Driver for VMware Cloud Director 1.6.1

Cause

This issue is caused by a missing command on the vcd-csi-plugin containers on the csi-vcd-nodeplugin pods.

Resolution

This issue is under review with VMware Engineering. Subscribe to receive an email when the article is updated.

Workaround

To workaround the issue use the previous 1.6.0 release of the Kubernetes Container Storage Interface Driver for VMware Cloud Director (CSI).

For new clusters, the Container Storage Interface (CSI) Component Version can be changed from 1.6.1 to the previous 1.6.0 release.
This can be done as per the documentation, Configure the VMware Cloud Director Container Service Extension Server Settings.
Example steps would be as follows:

Log into the Cloud Director Provider portal as a System Administrator.
Navigate to More > Kubernetes Container Clusters > CSE Management > Server Details.
Confirm that Component Versions > Container Storage Interface (CSI) currently shows version 1.6.1.
Click Update Server and ensure the default Update Configuration option is selected.
Click Next to go to Set Configuration Parameters and under CSE Server Components change the Container Storage Interface (CSI) Version from 1.6.1 to 1.6.0.
All other options can be left at their current settings.
Scroll to the bottom of the list and click Submit Changes.
Restart all CSE Servers to to apply the updated configuration.
New clusters will be created with CSI version 1.6.0 where this issue does not occur.

For existing clusters, the Container Storage Interface (CSI) Component Version can be downgraded from 1.6.1 to the previous 1.6.0 release.
This can be done by using the cluster component upgrade script to redeploy the cluster components with the Container Storage Interface (CSI) Component Version set to 1.6.0.

WARNING: This process assumes that all cluster components have already been upgraded to the latest release, ensure this is the case before proceeding.
This process will redeploy all the cluster components and not just CSI.
For more information on the cluster component upgrade script see the documentation, Upgrade Kubernetes Components in VMware Cloud Director Container Service Extension Clusters.

Example steps would be as follows:

Log into the Cloud Director Provider portal as a System Administrator, or into the Tenant portal for the Organization where the affected Kubernetes Container Cluster is located, and download the cluster's kubeconfig file.
For more information see the documentation, Manage Clusters, Download Kube Config.
Log into a client machine that meets the prerequisites in the cluster component upgrade script documentation:
- The machine must use Ubuntu GNU/Linux 20.04 operating system.
- Ensure kubectl is installed. For more information, see Install and Set Up kubectl on Linux.
- Ensure you install and operate Docker. For more information, see Install Docker Engine.
- Ensure the kubeconfig of the cluster is present on the machine at an accessible path.
Use the following command to set $HOME directory:

export HOME=<directory of choice>
Confirm $HOME directory is set correctly, use the following command to print the $HOME directory:

echo $HOME
Use the following command to create a folder structure for mounting and storing the cluster upgrade script content:

mkdir -p $HOME/cluster-upgrade-script
Navigate to the folder created, and use the following command to pull the cluster-upgrade-script image from http://projects.packages.broadcom.com:

cd $HOME/cluster-upgrade-script
docker pull projects.packages.broadcom.com/vmware-cloud-director/cluster-upgrade-script-airgapped:v0.1.6
After you pull the image, use docker to extract the image contents to the folder by creating a container and extracting it:

docker container create --name "temp_container" projects.packages.broadcom.com/vmware-cloud-director/cluster-upgrade-script-airgapped:v0.1.6
docker export "temp_container" | tar -C $HOME/cluster-upgrade-script -xvf - --strip-components 2
After the contents are extracted to the folder, it is safe to remove the temporary container:

docker container rm "temp_container"
After all the content is extracted to the main folder, open the directory, and update the permissions for the script:

cd $HOME/cluster-upgrade-script
chmod +x upgrade_cluster_components.sh
Open the upgrade_cluster_components.sh script to change the Container Storage Interface (CSI) Component Version from 1.6.1 to 1.6.0:

vi upgrade_cluster_components.sh
Locate the section # Set the versions of the target components and change the csi_version from 1.6.1 to 1.6.0:

Default:
csi_version="1.6.1"

Updated:
csi_version="1.6.0"
Save the changes to the upgrade_cluster_components.sh script.
Run the following command in the $HOME/cluster-upgrade-script folder using the kubeconfig file of the cluster where we wish to downgrade the Container Storage Interface (CSI) Component Version to 1.6.0 and the Broadcom image registry:

./upgrade_cluster_components.sh <kubeconfig_file_path> projects.packages.broadcom.com
After the script completes, confirm that pods can now mount the persistent volumes and enter a Running state.

Feedback

thumb_up Yes

thumb_down No