vVol queryComplianceResult API call replies with NotFound on a VM migrated from VMFS. No interruption to vVol operations seen.

Products

• VMware vSphere ESXi

Issue/Introduction

• VM migration to vVols occurs.  VM attributes are associated.  sps.log:
  
  • 2024-04-03T14:44:37.617Z [pool-4-thread-5] INFO  opId=1b66fb49-01 com.vmware.pbm.profile.impl.ProfileManagerImpl - Starting Timer: preProvisionProcess. Entity associations: EntityAssociations{entity = {key = vm-277414}, entityName = friendly_name_datastore, operation = MIGRATE, policyAssociations = [{entity = {key = vm-277414}, hub = {hubId = datastore-277159}, policySpec = {type = DefaultProfileSpec}, defaultPolicy = false}, {entity = {key = vm-277414:2000}, hub = {hubId = datastore-277159}, policySpec = {type = DefaultProfileSpec}, defaultPolicy = false}, {entity = {key = vm-277414:2001}, hub = {hubId = datastore-277159}, policySpec = {type = DefaultProfileSpec}, defaultPolicy = false}]}
    
    
• Post VM migration to vVol, reconfigEntityAssociations API is called to update DB data and compute compliance for the VM and its vmdks.

  • 2024-04-03T14:46:38.467Z [pool-4-thread-10] INFO  opId=1b66fb49-01 com.vmware.pbm.profile.impl.ProfileManagerImpl - Starting Timer: reconfigEntityAssociations. Ops: [DefaultProfileAssociateOp{entity = {key = vm-277414}, profileId = {uniqueId = ##-##-##-##-##}, diskEntity = null, replicationSpec = null, hub = {hubId = datastore-277159}}, DefaultProfileAssociateOp{entity = {key = vm-277414:2000}, profileId = {uniqueId = ##-##-##-##-##}, diskEntity = null, replicationSpec = null, hub = {hubId = datastore-277080}}, DefaultProfileAssociateOp{entity = {key = vm-277414:2001}, profileId = {uniqueId = ##-##-##-##-##}, diskEntity = null, replicationSpec = null, hub = {hubId = datastore-277080}}]

    2024-04-03T14:46:38.518Z [pool-4-thread-10] INFO  opId=1b66fb49-01 com.vmware.pbm.profile.impl.ProfileManagerImpl - Timer stopped: reconfigEntityAssociations, Time taken: 51 ms.

• Storage Policy compliance calculation follows.  SPS responds with the backing Id friendly name of friendly_name_datastore.  The VASA Provider queryComplianceResult API call returns a NotFound error.

  • 2024-04-03T14:46:38.541Z [pool-8-thread-1479] DEBUG opId=1b66fb49-01 com.vmware.spbm.domain.policy.Entity - Backing object id for vm-277414 : friendly_name_datastore

    2024-04-03T14:46:38.588Z [pool-34-thread-4] DEBUG opId=1b66fb49-01 httpclient.wire.content - >> "<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body><ns7:queryComplianceResult xmlns:ns7="http://com.vmware.vim.vasa/2.0/xsd"><ns7:compliance><ns2:generationId xmlns:ns2="http://compliance.policy.data.vasa.vim.vmware.com/xsd">0</ns2:generationId><ns2:objectId xmlns:ns2="http://compliance.policy.data.vasa.vim.vmware.com/xsd">friendly_name_datastore</ns2:objectId><ns2:profileId xmlns:ns2="http://compliance.policy.data.vasa.vim.vmware.com/xsd">##-##-##-##-##</ns2:profileId></ns7:compliance></ns7:queryComplianceResult></soapenv:Body></soapenv:Envelope>"

    2024-04-03T14:46:38.598Z [pool-34-thread-4] DEBUG opId=1b66fb49-01 httpclient.wire.content - << "l version="1.0" encoding="utf-8" ?><env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope"><env:Header/><env:Body><env:Fault><env:Code><env:Value>env:Receiver</env:Value></env:Code><env:Reason><env:Text xml:lang="en-US">NotFound</env:Text></env:Reason><env:Detail><ns2:NotFound xmlns:ns2="http://com.vmware.vim.vasa/2.0/xsd" xmlns:ns3="http://fault.vasa.vim.vmware.com/xsd" xmlns:ns4="http://policy.data.vasa.vim.vmware.com/xsd" xmlns:ns5="http://data.vasa.vim.vmware.com/xsd" xmlns:ns6="http://fault.replication.vasa.vim.vmware.com/xsd"/></env:Detail></env:Fault></env:Body></env:Envelope>"

    2024-04-03T14:46:38.599Z [pool-34-thread-4] DEBUG opId=1b66fb49-01 com.vmware.vim.sms.policy.PolicyManagerImpl - Timer stopped: queryComplianceResult, Time taken: 34 ms.
2024-04-03T14:46:38.599Z [pool-34-thread-4] ERROR opId=1b66fb49-01 com.vmware.sps.pbm.impl.LocalSMSServiceImpl - Exception occurred during compliance calculationcom.vmware.vim.vasa.NotFound: com.vmware.vim.vasa._4_0.NotFound: NotFound

• SPS updates the VM backing Id to have a object Id versus friendly name and retries the queryComplianceResult API response.  The expected compliant response is received.

  • 2024-04-03T14:46:38.601Z [pool-226-thread-2] DEBUG opId=1b66fb49-01 com.vmware.spbm.domain.policy.Profile - Executing getCompliance() for subjects: [EntitySubject{entity = {key = vm-277414:2000, objectType = virtualDiskId}, profile = {profileId = ##-##-##-##-##, name = VVol No Requirements Policy, profileCategory = REQUIREMENT, lastUpdatedTime = 2022-11-14T09:36:22.097Z, subProfiles = [], generationId = 0, lineOfService = null, resourceType = STORAGE}, storageId = naa.XXXXX, vmObject = null, vmInstanceUuid = null, isOnObjectBasedStorage = false, schemaNamespace = null, tagInfo = null, datastoreType = VMFS}, EntitySubject{entity = {key = vm-277414:2001, objectType = virtualDiskId}, profile = {profileId = ##-##-##-##-##, name = VVol No Requirements Policy, profileCategory = REQUIREMENT, lastUpdatedTime = 2022-11-14T09:36:22.097Z, subProfiles = [], generationId = 0, lineOfService = null, resourceType = STORAGE}, storageId = naa.XXXXX, vmObject = null, vmInstanceUuid = null, isOnObjectBasedStorage = false, schemaNamespace = null, tagInfo = null, datastoreType = VMFS}, EntitySubject{entity = {key = vm-277414, objectType = virtualMachine}, profile = {profileId = ##-##-##-##-##, name = VVol No Requirements Policy, profileCategory = REQUIREMENT, lastUpdatedTime = 2022-11-14T09:36:22.097Z, subProfiles = [], generationId = 0, lineOfService = null, resourceType = STORAGE}, storageId = vvol:##-##, vmObject = friendly_name_datastore, vmInstanceUuid = ##-##-##-##-##, isOnObjectBasedStorage = true, schemaNamespace = ##.##.VVOL, tagInfo = null, datastoreType = VVOL}]

    2024-04-03T14:46:48.655Z [pool-8-thread-1480] DEBUG opId=1b66fb49-01 com.vmware.spbm.domain.policy.Entity - Backing object id for vm-277414 : naa.#####

    2024-04-03T14:46:48.700Z [pool-34-thread-2] DEBUG opId=1b66fb49-01 httpclient.wire.header - >> "Content-Type: application/soap+xml; charset=UTF-8; action="urn:queryComplianceResult"[\r][\n]"

    2024-04-03T14:46:48.700Z [pool-34-thread-2] DEBUG opId=1b66fb49-01 httpclient.wire.content - >> "<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body><ns7:queryComplianceResult xmlns:ns7="http://com.vmware.vim.vasa/2.0/xsd"><ns7:compliance><ns2:generationId xmlns:ns2="http://compliance.policy.data.vasa.vim.vmware.com/xsd">0</ns2:generationId><ns2:objectId xmlns:ns2="http://compliance.policy.data.vasa.vim.vmware.com/xsd">naa.#####</ns2:objectId><ns2:profileId xmlns:ns2="http://compliance.policy.data.vasa.vim.vmware.com/xsd">##-##-##-##-##</ns2:profileId></ns7:compliance></ns7:queryComplianceResult></soapenv:Body></soapenv:Envelope>"

    2024-04-03T14:46:48.709Z [pool-34-thread-2] DEBUG opId=1b66fb49-01 httpclient.wire.content - << "<?xml version="1.0" encoding="utf-8" ?><env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope"><env:Header/><env:Body><ns5:queryComplianceResultResponse xmlns="http://compliance.policy.data.vasa.vim.vmware.com/xsd" xmlns:ns2="http://policy.data.vasa.vim.vmware.com/xsd" xmlns:ns3="http://capability.policy.data.vasa.vim.vmware.com/xsd" xmlns:ns4="http://lineofservice.data.vasa.vim.vmware.com/xsd" xmlns:ns5="http://com.vmware.vim.vasa/2.0/xsd"><ns5:return><checkTime>2024-04-03T14:46:48.702Z</checkTime><complianceStatus>compliant</complianceStatus><objectId>naa.#####</objectId><profileId>##-##-##-##-##</profileId><profileMismatch>false</profileMismatch></ns5:return></ns5:queryComplianceResultResponse></env:Body></env:Envelope>"

    2024-04-03T14:46:48.725Z [pool-9-thread-13112] INFO  opId=1b66fb49-01 com.vmware.pbm.profile.impl.AssociationServiceImpl - Timer stopped: UpdateComplianceOnAssociateTask, Time taken: 10207 ms.

Environment

VMware vSphere 7.x

VMware vSphere 8.x

Cause

• According to the vVol specification, the object Id should be a globally unique Id, hence friendly names should not be passed.  The NotFound response to the queryComplianceResult API call is expected from the VASA Provider.
• The retry mechanism leveraged by vCenter/SPS is appropriate for some situations where VM backing data may not be completely refreshed and updated prior to receiving the queryComplianceResult API call.

Resolution

During log reviews some partners (VASA Providers) have noticed the temporary deviation from using a globally unique Id and have inquired to determine if the behavior is a concern.  VMware Engineering has confirmed the above behavior is transparent to the user and should not inhibit vVol operations.