How to track vSAN objects state and changes when "nonavailability-related-incompliance" state is reported via Skyline Health
search cancel

How to track vSAN objects state and changes when "nonavailability-related-incompliance" state is reported via Skyline Health

book

Article ID: 401291

calendar_today

Updated On:

Products

VMware vSAN 8.x VMware vSAN 7.x

Issue/Introduction

This article provides details of object operation when an object is reported (not compliant) with its own storage policy due modification.

 

Details of objects in "nonavailability-related-incompliance" can be obtained using the below command:

# esxcli vsan debug object list --health=nonavailability-related-incompliance

 

Symptoms: 

  • /var/run/log/clomd.log of a host shows messages similar to:

 

2025-05-21T15:22:15.094Z info clomd[2099327] [Originator@6876] CLOM_PostWorkItem: Posted a work item opID:1804290321 for xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx group: xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx Type: CHANGE delay 0 (Success)

 

  • Taking closer look at operation [opID: 1804290321]  (Note the CHANGE type, this is not a repair object activity, i.e.: object did not lose any mirrors at this stage):

2025-05-21T15:22:15.094Z info clomd[2099327] [Originator@6876] CLOM_PostWorkItem: Posted a work item opID:1804290321 for xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx group: xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx Type: CHANGE delay 0 (Success)
2025-05-21T15:22:15.094Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMProcessWorkItem: Op CHANGE starts:1804290321
2025-05-21T15:22:15.096Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMReconfigure: Reconfiguring xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx workItem type CHANGE

 

  • vSAN object is reporting (not compliant): 

2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOM_VerifyPolicyComplianceVsan1: Obj xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx is not compliant, reason:0x10

 

  • Storage Policy Change from FTT1 to FTT1, with size increase, i.e.: inflation -> object did not lose availability and/or redundancy (mirrors) - it only became temporarily incompliant with original policy:

2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOM_VerifyPolicyComplianceVsan1: Current Policy
2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMLogConfigurationPolicy: Object size 102005473280 bytes with policy: (("stripeWidth" i1) ("capacity" (l0 l102005473280)) ("proportionalCapacity" (i0 i100)) ("hostFailuresToTolerate" i1) ("affinity" [ xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx]) ("storageType" "AllFlash") ("replicaPreference" "Performance"))
2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOM_VerifyPolicyComplianceVsan1: Target Policy
2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMLogConfigurationPolicy: Object size 273804165120 bytes with policy: (("stripeWidth" i1) ("hostFailuresToTolerate" i1) ("forceProvisioning" i0) ("spbmProfileId" "xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx") ("spbmProfileGenerationNumber" l+1) ("CSN" l425) ("SCSN" l227) ("spbmProfileName" "vSAN Default Storage Policy"))

 

  • Tracking the same operation a nature of change (expansion) is shown: 

2025-05-21T15:22:15.110Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMInPlaceExpandObject: Inplace Expansion of xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx from 102005473280 to 273804165120
2025-05-21T15:22:15.111Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMInPlaceExpandObject: Inplace Expansion for xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx finished with Success

 

  • Finally, the operation is completed with "status Success":

2025-05-21T15:22:15.117Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMReconfigure: exit: obj xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx transiantCapGenerated - total: 0, site1: 0, site2: 0, workItem type CHANGE configDelay 0 newConfigGenerated 1 status Success
2025-05-21T15:22:15.118Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOM_PublishResyncBytes: No more work for xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx (Success), reset queued resync bytes to 0
2025-05-21T15:22:15.118Z info clomd[2099327] [Originator@6876 opID=1804290321] CLOMProcessWorkItem: Op ends:1804290321

 

 

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Resolution

vSAN Skyline health in vCenter will reporting Object health warning with object in "nonavailability-related-incompliance":

This state in vSAN indicates that an object is not fully compliant with its storage policy, but it's still meeting the minimum availability requirements defined by the (NumberOfFailuresToTolerate) setting. Essentially, the object might be partially or temporarily out of compliance due to policy changes or other factors, but it's still able to maintain its data's availability. This is not a critical error, and the object is not at risk. 

  • This state can occur when:
    • A policy is being changed on-the-fly, and the object hasn't fully been reconfigured to meet the new requirements. 
    • There are temporary resource limitations, preventing the object from being fully compliant. 
    • A failure occurred, but vSAN was able to tolerate it, and the object is being re-protected. 
Powered by Wolken Software