Error "The object vim.cluster.VsanlscsiTargetSystem:vsan-iscsi-target-system has already been deleted" - vSAN iSCSI Target Service health error
search cancel

Error "The object vim.cluster.VsanlscsiTargetSystem:vsan-iscsi-target-system has already been deleted" - vSAN iSCSI Target Service health error

book

Article ID: 449818

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:

  • The vCenter UI frequently displays inconsistent status information for the vSAN iSCSI Target Service. In some instances, the service appears as Disabled, but after refreshing the page, it shows as Enabled.
  • The vSAN iSCSI Target Service initially shows as healthy, but refreshing the tab causes it to enter an unhealthy state with the error:

    "The object 'vim.cluster.VsanlscsiTargetSystem:vsan-iscsi-target-system' has already been deleted or has not been completely created".



  • The vSAN Skyline Health frequently reports health check error on the Home Object.



  • Proceeding by clicking troubleshoot option for the error "Home object" confirms that "vSAN reports this issue when the vSAN iSCSI target service configuration file stored in the home object directory is not present or cannot be parsed."
  • Skyline Health suggests checking the following diagnostic steps across all cluster nodes:

      • Check if vSAN iSCSI target service is enabled and all service daemons are running properly on all ESXi host through 'Service runtime status' health
      • Check if the configure file exists from the directory /vmfs/volumes/vsanDatastore/.iSCSI-CONFIG/etc/vit.conf in all ESXi hosts.
      • Check if there is any invalid characters in the configuration file /vmfs/volumes/vsanDatastore/.iSCSI-CONFIG/etc/vit.conf by comparing the back up file from
        /var/run/log/vit.conf.backup.

  • Navigate to vSAN Cluster > Configure > iSCSI Targets. Note down the I/O Owner Host name.



    In this example, I/O Owner Host is "Host -3"

Environment

  • VMware vSAN 7.x
  • VMware vSAN 8.x
  • VMware vSAN 9.x

Cause

  • This issue occurs due to a stale file lock on the "vit.conf" configuration file. A non-I/O owner ESXi host in the cluster holds an active Exclusive lock on the file, preventing the active I/O owner host from reading or updating the configuration.

  • On the vCenter Server, the /var/log/vmware/vsan-health/vmware-vsan-health-service.log log file confirms that attempts to query the vSAN iSCSI Target Service configuration fail. 

    YYYY-MM-DDTHH:MM.SSSZ ERROR vsan-mgmt[#####] [VsanVcIscsiTargetSystemImpl::EntityInHost opID=###-#######-####] Error query host ######## for operation GetIscsiTargetServiceConfig
    Traceback (most recent call last):
      File "bora/vsan/vit/vpxd/pyMoVsan/VsanVcIscsiTargetSystemImpl.py", line 693, in EntityInHost
      File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 601, in <lambda>
      File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 376, in _InvokeMethod
    PyCppVmomi.vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) {
      msg = 'Received SOAP response fault from [<<cs p:######, PIPE:/var/run/envoy-hgw/hgw-pipe>, /hgw/host-##/vsan>]: getIscsiTargetServiceConfig\n',
      obj = 'vim.cluster.VsanIscsiTargetSystem:vsan-iscsi-target-system'
    }
    YYYY-MM-DDTHH:MM.SSSZ ERROR vsan-mgmt[#####] [VsanVcIscsiTargetSystemImpl::_GetIscsiTargetServiceConfig opID=###-#######-####] Exception (vmodl.fault.ManagedObjectNotFound) {
      msg = 'Received SOAP response fault from [<<cs p:######, PIPE:/var/run/envoy-hgw/hgw-pipe>, /hgw/host-##/vsan>]: getIscsiTargetServiceConfig\n',
      obj = 'vim.cluster.VsanIscsiTargetSystem:vsan-iscsi-target-system'
    } occured when GetIscsiTargetServiceConfig in host, refer to cluster state
    Traceback (most recent call last):
      File "bora/vsan/vit/vpxd/pyMoVsan/VsanVcIscsiTargetSystemImpl.py", line 1005, in _GetIscsiTargetServiceConfig
      File "bora/vsan/vit/vpxd/pyMoVsan/VsanVcIscsiTargetSystemImpl.py", line 854, in _QueryEntityInHost
      File "bora/vsan/vit/vpxd/pyMoVsan/VsanVcIscsiTargetSystemImpl.py", line 719, in EntityInHost
      File "bora/vsan/vit/vpxd/pyMoVsan/VsanVcIscsiTargetSystemImpl.py", line 693, in EntityInHost
      File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 601, in <lambda>
      File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 376, in _InvokeMethod
    PyCppVmomi.vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) {
      msg = 'Received SOAP response fault from [<<cs p:00007f1ee0833510, PIPE:/var/run/envoy-hgw/hgw-pipe>, /hgw/host-##/vsan>]: getIscsiTargetServiceConfig\n',
      obj = 'vim.cluster.VsanIscsiTargetSystem:vsan-iscsi-target-system'
    }
  • Access "vit.conf" file on all cluster nodes. On all or few hosts, accessing this file fails with a resource busy error:

    less /vmfs/volumes/vsanDatastore/.iSCSI-CONFIG/etc/vit.conf
    vit.conf: Device or resource busy

  • Identify the ESXi host holding the Exclusive lock by running "vmfsfilelockinfo" command on the affected hosts:

    vmfsfilelockinfo -p "/vmfs/volumes/vsanDatastore/.iSCSI-CONFIG/etc/vit.conf"

    vmfsfilelockinfo Version 2.0
    Looking for lock owners on "vit.conf"
    "vit.conf" is locked in Exclusive mode by host having mac address ['##:##:##:##:##:##']
    Trying to make use of Fault Domain Manager
    Found 6 ESX hosts using Fault Domain Manager.

    Searching on Host Host - 1
    Searching on Host Host - 2
    MAC Address : ##:##:##:##:##:##'

    Host owning the lock on file is Host - 2, lockMode : Exclusive
    Total time taken : 0.2545278499601409 seconds.

    In this example, Host - 2 is holding the exclusive lock (But the current I/O Owner is HOST - 3)

Resolution

Restart the "vitd" and "vsanmgmtd" services on the host (In this example, Host - 2) holding the stale lock.

  1. /etc/init.d/vitd restart
  2. /etc/init.d/vsanmgmtd  restart
Powered by Wolken Software