When many Persistent Volumes by file share are deleted, vsan health can crash.

Products

• VMware vCenter Server

Issue/Introduction

• This article helps users to prevent VSAN health crash when many file shares are deleted concurrently.
• Vsan health crashes and /var/core/core.vsanvcmgmtd-wor.**** can be found, some file shares are not deleted.

Environment

VMware vCenter Server 8.0 U3 and later

Cause

File share delete request goes through several components, the latency of each file share delete request can be high.

When Kubeletes deletes many PVs backed by file shares concurrently, many concurrent CNS tasks are pending in the vsan health service thread pool, once the thread pool runs out of resource, vsan health will panic and crash, and some file shares are not deleted.

Resolution

Broadcom is working towards a permanent fix for this issue.

Workaround: Update the value for 'maxConcurrentFileServiceRequests' from 0 (meaning unlimited) to 32 or lesser value in the VsanVcMgmtConfig.xml file.

• Login to vCenter server via SSH

• Update 'maxConcurrentFileServiceRequests' to 32 by navigating to /usr/lib/vmware-vsan/:

• • cd /usr/lib/vmware-vsan/
  • sed -i '152s/0/32/' VsanVcMgmtConfig.xml

• Verify that the value has been updated to 32:

  cat VsanVcMgmtConfig.xml | grep maxConcurrentFileServiceRequests

• Restart vsan-health service:

service-control --restart vsan-health