Scoreboard file are not removed when VM is deleted from disk.
search cancel

Scoreboard file are not removed when VM is deleted from disk.

book

Article ID: 402301

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • DCA backup application was creating automated backups. Issue occurred without any further intervention.
  • When the backup app will delete a virtual machine, its folder remains on the datastore including the scoreboard file. 
  • The host daemon log, /var/run/log/hostd.log, contains entries similar for snapshot deleting files:

    [timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/vmware.log'.
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/###-#########.vmsn'.
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/########-####-####-####-############.vmem.WRITELOCK'.
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/###-########.vmem.WRITELOCK'.
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/###-#########.vmem.WRITELOCK'.
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/vm_name.vmsd~'.
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/vm_name.vmsd.lck'.
[timestamp] Db(167) Hostd[2098672]: [Originator@6876 sub=Hostsvc.AssignableHardwareProvider opID=#####-####-#### sid=5237c1f4 user=root] AH GetBinding: 0
[timestamp] In(166) Hostd[2098672]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] IOFilterInfoImpl: #####-######## Id: VMW_spm_1.0.0, localId: spm
[timestamp] In(166) Hostd[2098672]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] IOFilterInfoImpl: #####-######## Id: VMW_vmwarevmcrypt_1.0.0, >localId: vmwarevmcrypt
[timestamp] In(166) Hostd[2098672]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] IOFilterInfoImpl: #####-######## Id: VMW_vmwarelwd_4.0.0, localId: >vmwarelwd
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/vm_name.vmsd'.
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/vm_name.vmx~'.
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/vm_name.vmx.lck'.
[timestamp] Db(167) Hostd[0000000]: [Originator@6876 sub=AdapterServer opID=#####-####-#### sid=5237c1f4 user=root] New request: target='vmodl.query.PropertyCollector:ha->########-#########', method='retrievePropertiesEx', session='########-####-####-####-############'
[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/vm_name.vmx'.
hostd.0:[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteFile deleted >'/vmfs/volumes/########-########-####-############/vm_name/vm_name.vmx'.

     

  • In the same host file we can see the Scoreboard file added: 
    hostd.0:[timestamp] In(166) Hostd[2098679]: [Originator@6876 sub=VMXStatsProvider(0000007edcd97ad0)] Add VM '22' with /vmfs/volumes/########-########-####->6c92cf04baf2/vm_name/<span style="color: rgb(230, 126, 35);">vm_name.scoreboard

     

  • In sequence there is the error message for not delete the VM folder:  
    hostd.0:[timestamp] In(166) Hostd[0000000]: [Originator@6876 sub=Libs opID=#####-####-#### sid=5237c1f4 user=root] SNAPSHOT: SnapshotDeleteVMInt: Couldn't remove >directory '/vmfs/volumes/########-########-####-############/vm_name'.

     

  • Message of the folder was completed deleted: 
    hostd.0:[timestamp] Db(167) Hostd[0000000]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-####-############/vm_name/vm_name.vmx opID=#####-####-#### >sid=5237c1f4 user=root] Virtual machine delete completed.


Environment

vSphere ESXi 8.0 Update 3e build 24674464

Cause

The issue occurs when a snapshot is taken before powering on a VM, this results in the snapshot not capturing the scoreboard file, which is only created after the VM is powered on and added to the VM's config file. When the VM is later restored from this snapshot, the scoreboard entry is removed from the config, leaving the actual file undeleted upon VM deletion. This leftover file blocks deletion of the VM folder.

 

Resolution

Workaround: Power on the VM before taking the snapshot.

Powered by Wolken Software