Shared vSAN witness component utilization has reached a critical threshold
search cancel

Shared vSAN witness component utilization has reached a critical threshold

book

Article ID: 438236

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms : 

  • vSAN Skyline health alert "Cluster And Host Component Utilization" reported for High Witness Component utlization. 



  • Physical Data Component Utilization on the Witness Host is reaching 99% (907 of 1000) 

[root@Lab-Shared-Witness-Host:~] vsish -e get /vmkModules/vsanbase/vsanlimits/componentLimitPerSharedWitnessCluster
1000

  • vSAN witness version is 8.0 P05 (ESXi 8.0 Update 3e - Build 24674464) or higher -- which is having the fix for Witness Component issue.

    Refer KB # 382735 for more details on the Witness Component issue.

 

Environment

VMware vSAN 8.0 Update 3 P05 or higher 

Cause

The advanced configuration parameter /VSAN/MaxWitnessClusters is set to its default value of 0 (which defaults to 64 clusters). vSAN calculates the per-cluster component limit by dividing the total witness capacity by the maximum number of supported clusters (e.g., 64,000 / 64 = 1,000). This mathematical cap prevents vSAN shared witness from utilizing more than 1,000 components, despite available global resources.

Cause Justification :

  • The vsish output for /config/VSAN/intOpts/MaxWitnessClusters confirms a current value of 0.
     
    • Global Capacity: 64,000 components.
    • Default Cluster Cap: 64 clusters.
    • Effective Limit: 1,000 components per cluster.

[root@Lab-Shared-Witness-Host:~] vsish -e get /config/VSAN/intOpts/MaxWitnessClusters
Vmkernel Config Option {
   Default value:0
   Min value:0
   Max value:64
   Current value:0
   hidden config option:0
   Description:Max number of clusters on a witness node (0: default to 64 clusters)
   Host specific config option:0
   Exclude option from config manager:0
   Option update requires reboot:0
   Option update requires maintenance mode:0

  • localcli vsan debug limit get shows "Component Limit Health" as green globally, confirming the issue is isolated to the per-cluster software ceiling, not physical host exhaustion.

[root@Lab-Shared-Witness-Host:~]  localcli vsan debug limit get

limitList:
   Component Limit Health: green
   Disk Free Space Health: green
   Free Components: 19906
   Lowest Free Disk Space: 93
   Max Components: 21792
   Total Disk Space: 375801249792
   Total Disk Space (GB): 349.99 GB
   Used Disk Space: 28294690897
   Used Disk Space (GB): 26.35 GB

Resolution

  • Increase the per-cluster component limit by reducing the MaxWitnessClusters value on the shared witness appliance. By lowering the maximum number of supported clusters, vSAN reallocates more component "slots" to each cluster.

Note : We have taken an example of 20 vSAN Cluster, based on the below example choose the value based on the business requirement i.e. Min value:0  and  Max value:64

# Syntax : esxcli system settings advanced set -o /VSAN/MaxWitnessClusters -i <value>

# Command : esxcli system settings advanced set -o /VSAN/MaxWitnessClusters -i 20

  • This change is non-disruptive and does not require a reboot. This will increase the per-cluster limit from 1,000 to approximately 2,145 components.

  • Lowering the max cluster count allows the witness to support larger individual clusters.

  • Lab test results confirms that changing the value to 20 immediately updates the componentLimitPerSharedWitnessCluster from 1000 to 2145. This provides the necessary headroom for shared vSAN witness appliance to expand its component count, clearing the Skyline Health alert.

  • Refer Below lab test results for more details. 

Note : The following example 20 vSAN Cluster is updated for example. 

Before Applying the fix the componentLimitPerSharedWitnessCluster is 1000.

[root@Lab-Shared-Witness-Host:~] vsish -e get /vmkModules/vsanbase/vsanlimits/componentLimitPerSharedWitnessCluster
1000

The MaxWitnessClusters for shared witness appliance is default value of zero which means it will support upto 64 vSAN clusters. 

[root@Lab-Shared-Witness-Host:~] vsish -e get /config/VSAN/intOpts/MaxWitnessClusters
Vmkernel Config Option {
   Default value:0
   Min value:0
   Max value:64
   Current value:0
   hidden config option:0
   Description:Max number of clusters on a witness node (0: default to 64 clusters)
   Host specific config option:0
   Exclude option from config manager:0
   Option update requires reboot:0
   Option update requires maintenance mode:0
}

After Applying the fix the componentLimitPerSharedWitnessCluster is 2145.

[root@Lab-Shared-Witness-Host:~] esxcli system settings advanced set -o /VSAN/MaxWitnessClusters -i 20

[root@Lab-Shared-Witness-Host:~] vsish -e get /config/VSAN/intOpts/MaxWitnessClusters
Vmkernel Config Option {
   Default value:0
   Min value:0
   Max value:64
   Current value:20
   hidden config option:0
   Description:Max number of clusters on a witness node (0: default to 64 clusters)
   Host specific config option:0
   Exclude option from config manager:0
   Option update requires reboot:0
   Option update requires maintenance mode:0
}
[root@Lab-Shared-Witness-Host:~] vsish -e get /vmkModules/vsanbase/vsanlimits/componentLimitPerSharedWitnessCluster
2145
[root@Lab-Shared-Witness-Host:~] 

 

Powered by Wolken Software