VKS Windows Applications Failing to Access Windows File Shares

Products

• VMware vSphere Kubernetes Service
• Tanzu Kubernetes Runtime

Issue/Introduction

While connected to the VKS cluster context where the affected Windows application pods are failing:

• Describing a pod reliant on csi-smb driver for its volume management, shows the following error:
  

  MountVolume.SetUp failed for volume <volume name> output kubernetes.io/csi: mounter.SetUpAt failed to check for STAGE_UNSTAGE_VOLUME capability: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\smb.csi.ks.io\\\\csi.sock: connect: No connection could be made because the target machine actively refused it."

   

• One or more csi-smb-node pods are in CrashLoopBackOff or have high, recent Restart counts:
  

  kubectl get pods -n kube-system -o wide

   

• Viewing the logs when one of these csi-smb-node pods crashed shows a stack trace containing the following error at the beginning, indicating that the pod reached or exceeded its memory limit:
  

  kubectl logs -n kube-system <csi-smb-node-pod>
  
  IMMDD HH:MM:SS.sssss    #### safe_mounter_host_process_windows.go:69] begin to mount \\<folder path> on c:\var\lib\kubelet\plugins\kubernetes.io\csi\smb.csi.k8s.io\<id>\globalmount
  runtime: VirtualAlloc of #### bytes failed with errno=1455
  fatal error: out of memory

 

Environment

vSphere Supervisor

Nodes using Windows OS

Pods using csi-smb driver for volume management

Cause

The csi-smb-node pods managing volumes through SMB are hitting or exceeding its configured memory limits.

Resolution

Increase the memory limit on the csi-smb-node daemonset which controls and manages the configuration for the csi-smb-node pods.

1. Connect into the VKS cluster context where the csi-smb-nodes are crashing
   
   
2. Locate the daemonset for the csi-smb-nodes:
   

   kubectl get daemonset -n kube-system

    

3. Edit the daemonset to increase the smb container's memory limit:
   

   kubectl edit daemonset -n kube-system <csi-smb-node daemonset>
   
   Containers:
   ...
      smb:
          Image: <smb image>
          State:   Running
          Limits:
               memory: <CHANGE ME>

    

4. Confirm that the csi-smb-node pods recreate successfully:
   

   kubectl get pods -n kube-system -o wide

    

5. Check that the smb driver reliant pods reach Running state:
   

   kubectl get pods -n <namespace for smb driver reliant pods>

    

6. If the csi-smb-node pods continue to crash:
   
   
   1. Confirm that the above daemonset memory limit changes did not revert:
      

      kubectl describe daemonset -n kube-system <csi-smb-node daemonset>

       

   2. Check if you need to increase the memory limit to a higher value on the daemonset.
      
      
      
   3. To be thorough, you can describe the csi-smb-node pods to confirm that the daemonset changes propagated:
      

      kubectl describe pod -n kube-system <csi-smb-node pod>

Additional Information

The csi-smb driver and daemonset are not owned by VMware by Broadcom.

We recommend that questions or concerns on its configuration should be brought up to the csi-smb maintainers and Microsoft Windows.