How to track vSAN objects state and changes when "nonavailability-related-incompliance" state is reported via Skyline Health
search cancel

How to track vSAN objects state and changes when "nonavailability-related-incompliance" state is reported via Skyline Health

book

Article ID: 401291

calendar_today

Updated On:

Products

VMware vSAN 8.x VMware vSAN 7.x

Issue/Introduction

This article provides details of object operation when an object is reported (not compliant) with its own storage policy due modification and you see the below messages in the logs. 

Symptoms: 

  • /var/run/log/clomd.log of a host shows messages similar to:

2025-05-21T15:22:15.094Z info clomd[2099327] [Originator@6876] CLOM_PostWorkItem: Posted a work item opID:1804290321 for ##########-####-####-####-######## group: ##########-####-####-####-######## Type: CHANGE delay 0 (Success)

  • Taking closer look at operation [opID: 1804290321]  (Note the CHANGE type, this is not a repair object activity, i.e.: object did not lose any mirrors at this stage):

2025-05-21T15:22:15.094Z info clomd[2099327] [Originator@6876] CLOM_PostWorkItem: Posted a work item opID:1804290321 for ##########-####-####-####-######## group: ##########-####-####-####-######## Type: CHANGE delay 0 (Success)
2025-05-21T15:22:15.094Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMProcessWorkItem: Op CHANGE starts:1804290321
2025-05-21T15:22:15.096Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMReconfigure: Reconfiguring ##########-####-####-####-######## workItem type CHANGE

  • vSAN object is reporting (not compliant): 

2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOM_VerifyPolicyComplianceVsan1: Obj ##########-####-####-####-######## is not compliant, reason:0x10

  • Storage Policy Change from FTT1 to FTT1, with size increase, i.e.: inflation -> object did not lose availability and/or redundancy (mirrors) - it only became temporarily incompliant with original policy:

2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOM_VerifyPolicyComplianceVsan1: Current Policy
2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMLogConfigurationPolicy: Object size 102005473280 bytes with policy: (("stripeWidth" i1) ("capacity" (l0 l102005473280)) ("proportionalCapacity" (i0 i100)) ("hostFailuresToTolerate" i1) ("affinity" [ ##########-####-####-####-######## ##########-####-####-####-######## ##########-####-####-####-########]) ("storageType" "AllFlash") ("replicaPreference" "Performance"))
2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOM_VerifyPolicyComplianceVsan1: Target Policy
2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMLogConfigurationPolicy: Object size 273804165120 bytes with policy: (("stripeWidth" i1) ("hostFailuresToTolerate" i1) ("forceProvisioning" i0) ("spbmProfileId" "##########-####-####-####-########") ("spbmProfileGenerationNumber" l+1) ("CSN" l425) ("SCSN" l227) ("spbmProfileName" "vSAN Default Storage Policy"))

  • Tracking the same operation a nature of change (expansion) is shown: 

2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMInPlaceExpandObject: Inplace Expansion of ##########-####-####-####-######## from 102005473280 to 273804165120
2025-05-21T15:22:15.111Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMInPlaceExpandObject: Inplace Expansion for ##########-####-####-####-######## finished with Success

  • Finally, the operation is completed with "status Success":

2025-05-21T15:22:15.117Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMReconfigure: exit: obj ##########-####-####-####-######## transiantCapGenerated - total: 0, site1: 0, site2: 0, workItem type CHANGE configDelay 0 newConfigGenerated 1 status Success
2025-05-21T15:22:15.118Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOM_PublishResyncBytes: No more work for ##########-####-####-####-######## (Success), reset queued resync bytes to 0
2025-05-21T15:22:15.118Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMProcessWorkItem: Op ends:1804290321

Another symptom is shown in the usage of 'Force Provisioning' & 'Thick Provisioning' together that can cause the kind of problem that will be experienced by the vSAN objects not being compliant with the storage policy. 

esxcli vsan debug object list --health = nonavailability-related-incompliance

Object UUID: c345ff67-1081-31ba-925a-############
   Version: 15
   Health: nonavailability-related-incompliance
   Owner: host-1.broadcom.org
   Size: 250.00 GB
   Used: 80 GB
   Used 4K Blocks: 80 GB
   Policy: 
      stripeWidth: 1
      cacheReservation: 0
      proportionalCapacity: 100
      hostFailuresToTolerate: 0
      affinity: ['6789decc-c0ea-597c-656f-############']
      forceProvisioning: 1
      affinityMandatory: 1
      spbmProfileId: 0781f51e-829d-47e0-8602-############
      spbmProfileGenerationNumber: 1
      storageType: Allflash
      migrateForDecom: 1
      iopsLimit: 0
      checksumDisabled: 0
      CSN: 1245
      spbmProfileName: Stripe1_Max0_Thick_0%_Force_Affinity
      locality: HostLocal

 

   Configuration: 
      
      Component: c345ff67-1081-31ba-925a-############
        Component State: ACTIVE,  Address Space(B): 268435456000 (250.00GB),  Disk UUID: 52ddc02a-5fba-9cb2-d2d6-############,  Disk Name: t10.NVMe____Dell_Ent_NVMe_CM6_RI_3:2
        Votes: 1,  Capacity Used(B): 273841910000 (255.04GB),  Physical Capacity Used(B): 66580381696 (62.01GB),  Total 4K Blocks Used(B): 66498410000 (61.93GB),  Host Name: host-1.broadcom.org

 

   Type: vdisk
   Path: /vmfs/volumes/vsan:################-################/########-####-####-####-############/virtualmachine.vmdk (Exists)
   Group UUID: ########-####-####-####-############
   Directory Name: N/A

 

The Mismatch (Thick vs. Thin): The active storage policy applied to the objects above mandates proportionalCapacity: 100, which means 100% Object Space Reservation (Thick Provisioning). However, the physical space actually used on the disks is much smaller than the allocated size. For example, object ########-####-####-####-############ is allocated for 250.00 GB, but its physical capacity used is only 49.62 GB.

 

The Affinity Bottleneck: The policy dictates hostFailuresToTolerate: 0 combined with a strict host affinity rule, meaning the entire object is pinned to one specific ESXi host without any replicas.

 

The Trigger: Because multiple 128 GB and 250 GB objects are pinned to individual hosts, those specific hosts do not have enough contiguous free space to fully inflate the disks to 100% Thick.

 

The Override: The objects are kept alive and functional because the policy has forceProvisioning: 1 enabled. This tells vSAN to deploy the disk as Thin provisioned to keep the VM running, but continuously flags it as non-compliant because it cannot fulfill the 100% Thick requirement.

Environment

VMware vSAN  

 

Cause

vSAN Skyline health in vCenter will reporting Object health warning with object in "nonavailability-related-incompliance":

This state in vSAN indicates that an object is not fully compliant with its storage policy, but it's still meeting the minimum availability requirements defined by the (NumberOfFailuresToTolerate) setting. Essentially, the object might be partially or temporarily out of compliance due to policy changes or other factors, but it's still able to maintain its data's availability. This is not a critical error, and the object is not at risk. 

This state can occur when:

    • A policy is being changed on-the-fly, and the object hasn't fully been reconfigured to meet the new requirements. 
    • There are temporary resource limitations, preventing the object from being fully compliant. 

A failure occurred, but vSAN was able to tolerate it, and the object is being re-protected. 

Resolution

When a storage policy is changed on the fly, vSAN Distributed Object Manager (DOM) applies these changes to the objects which can temporarily increase the object sizes to comply with the new RAID policy. nonavailability-related-incompliance is a temporary alert while the new modified storage policy is in effect, objects will revert to their normal RAID policy when vSAN completes the policy change.

You have two paths to resolve the incompliance, depending on how you actually want these disks provisioned:

Option 1: Revert the Policy to Thin Provisioning (Recommended)

If these VMs were intended to be Thin provisioned (as the policy name implies):

  1. In the vSphere Client, go to Policies and Profiles > VM Storage Policies.

  2. Edit the Stripe1_Max0_Thick_0%_Force_Affinity policy.

  3. Change the Object Space Reservation rule from 100% back to 0%.

  4. Save the policy and choose to apply it to all compliant virtual machines. The 138 objects will immediately shift to a Healthy state because vSAN no longer has to try inflating them.

Option 2: Free Up Space on the Pinned Hosts

If these VMs genuinely require 100% Thick Provisioning, the targeted affinity hosts lack the raw capacity to inflate them.

  1. Identify the specific hosts pinned in the affinity rules  

  2. Storage vMotion other workloads off those specific hosts to free up backend capacity.

  3. Once enough space is cleared, vSAN will automatically inflate the disks to 100% and clear the incompliance alarms.

Additional Information

vSAN Health Service - Data Health – vSAN Object Health

vSAN objects in "nonavailability-related-incompliance" state

Because vSAN is an object-based storage system (rather than traditional LUNs or volumes), every VMDK, swap file, and VM home directory is an independent object. The esxcli vsan debug object list command output tells you exactly what the object is, how healthy it is, what rules govern it, and where its physical data components live across your servers.

Here is a breakdown of what each section and key field represents:

Core Identity & Health

  • Object UUID: The unique cryptographic identifier for this specific object.

  • Version: The on-disk format version of the object (e.g., Version 15 correlates to vSAN 8.0).

  • Health: The current operational state (e.g., healthy, inaccessible, nonavailability-related-incompliance).

  • Owner: The specific ESXi host currently designated as the "DOM Owner," responsible for coordinating I/O and locking operations for this object.

  • Size: The logical, provisioned size of the object (how big the VM thinks the disk is).

  • Used / Used 4K Blocks: The actual physical capacity currently consumed on the backend drives.

Policy (Storage Policy Based Management Rules) This section shows the active VM Storage Policy rules currently governing how the object is protected and provisioned.

  • hostFailuresToTolerate (FTT): The primary redundancy rule. FTT=1 means the object is mirrored or uses RAID-5 to survive one host/disk failure. FTT=0 means there is no redundancy.

  • proportionalCapacity: The Object Space Reservation. 0 means standard Thin Provisioning. 100 means Thick Provisioning (reserving 100% of the space on the physical drives).

  • forceProvisioning: If set to 1, vSAN is allowed to instantiate the object and keep it online even if the cluster currently lacks the resources to fully satisfy the policy (which usually triggers the nonavailability-related-incompliance alarm).

  • stripeWidth: The number of physical disks across which the object's data is striped (for performance).

  • affinity / affinityMandatory: Rules used to pin or restrict the object to specific ESXi hosts (often used in edge cases or 2-node clusters).

  • spbmProfileName: The human-readable name of the Storage Policy assigned in vCenter.

  • CSN (Configuration Sequence Number): An internal counter that tracks how many times the object's layout has been modified or rebuilt.

Configuration (The Physical Layout) This section maps the logical object down to the physical hardware.

  • Component: An object is made up of multiple components (e.g., two data components and a witness component for a RAID-1 mirror). This lists every component's UUID, the specific physical disk (Disk UUID / Disk Name) it lives on, and the host it resides on.

  • Component State: Shows if the component is ACTIVE (healthy), ABSENT (host offline or network dropped), or STALE (out of sync).

  • Votes: The number of quorum votes this specific component contributes to the object. An object must maintain more than 50% of its total votes across all components to remain accessible.

Context & Location

  • Type: Identifies the object's function. Common types are vdisk (a virtual hard drive / VMDK), vmnamespace (the VM's home folder containing .vmx and log files), or vmswap (memory swap file).

  • Path: The logical file path where this object is accessible on the vSAN datastore.

  • Group UUID: The UUID of the vmnamespace object that this specific disk or swap file belongs to. This groups all of a single VM's objects together.

Powered by Wolken Software