"ESXi tracking filter not available" error when consolidating snapshots

Products

• VMware Cloud Foundation

Issue/Introduction

Provide a brief overview of the article to include information that the reader will be interested in knowing about the article and a quick gist of what the article is all about

This articles provides a workaround to address snapshot consolidation issue due to failure in loading CBT module during ESXi boot process.

• Snapshot consolidation fails to complete post upgrade of ESXi version to 7.0 U1 using DELL Custom Image
• Virtual Machine logs and Hostd logs will report failure in opening change tracking file (CTK) resulting with consolidation error "ESXi tracking filter not available"
• You will see following errors in virtual machine log file </vmfs/volumes/<Datastore UUID>/<Virtual Machine Name>/vmware.log

2020-10-24T10:02:18.629Z| vcpu-0| A000: ConfigDB: Setting scsi0:1.redo = ""
2020-10-24T10:02:18.629Z| vcpu-0| I005: DISK: OPEN scsi0:1 '/vmfs/volumes/datastore1/########-####-########d07c/Test_1-000002.vmdk' persistent R[]
2020-10-24T10:02:18.631Z| vcpu-0| I005: AIOGNRC: Failed to open '/vmfs/devices/disks/########-####-########d07c' : Could not find the file (60003) (0x2013).
2020-10-24T10:02:18.631Z| vcpu-0| I005: OBJLIB: Device '/vmfs/devices/disks/########-####-########d07c' not present, retrying to create one.
2020-10-24T10:02:18.641Z| vcpu-0| I005: DISKLIB-LIB_BLOCKTRACK   : Resuming from change tracking info file /vmfs/volumes/datastore1/########-####-########d07c/Test_1-000002-ctk.vmdk.
2020-10-24T10:02:18.645Z| vcpu-0| I005: DISKLIB-CTK   : Could not open change tracking file "/vmfs/volumes/datastore1/########-####-########d07c/Test_1-000002-ctk.vmdk": Change tracking invalid
2020-10-24T10:02:18.651Z| vcpu-0| I005: DISKLIB-CTK   : Change tracking invalid; reinitializing.
2020-10-24T10:02:18.683Z| vcpu-0| I005: ConsolidateEnd: Snapshot consolidate complete: ESXi tracking filter not available (5).

• You will following entries in var/run/log/hostd.log

2020-10-26T15:48:44.261Z verbose hostd[3457085] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/datastore1/########-####-########d07c/Test.vmx opID=kg4v90cf-4586521-auto-2qaze-h5:70279734-6-78-1f35 user=vpxuser:TRIODOS\admincklok] Removeallsnapshots received. Consolidate: true
2020-10-26T15:48:44.262Z info hostd[3457085] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/datastore1/########-####-########d07c/Test.vmx opID=kg4v90cf-4586521-auto-2qaze-h5:70279734-6-78-1f35 user=vpxuser:TRIODOS\admincklok] State Transition (VM_STATE_ON -> VM_STATE_REMOVEALL_SNAPSHOT)
2020-10-26T15:48:44.359Z verbose hostd[3457085] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/datastore1/########-####-########d07c/Test.vmx opID=kg4v90cf-4586521-auto-2qaze-h5:70279734-6-78-1f35 user=vpxuser:TRIODOS\admincklok] Consolidate disks after snapshot removal.
2020-10-26T15:48:45.415Z verbose hostd[2102328] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/datastore1/########-####-########d07c/Test.vmx] Consolidate Disks message: Consolidation failed for disk node 'scsi0:0': ESXi tracking filter not available.

• Consolidation failed for disk node 'scsi0:1': ESXi tracking filter not available.

• An error occurred while consolidating disks: ESXi tracking filter not available.

Environment

VCF

Cause

Include a high-level statement that explains the root cause of the issue to provide context for the customer. Ensure the cause explains only what is relevant to the issue.

CBT Module fails to load during ESXi boot post upgrade is performed.

Note: Software FCoE is no longer supported in vSphere 7.

Resolution

This is a known issue affecting VMware ESXi 7.0 Update 1.

You can implement the workaround or Upgrade the ESXi to 7.0 U2 or to latest release in 7.0 series 

Note: Consolidation of delta disk can be accomplished when a virtual machine is in a powered off state. 

Manually load CBT Module:

• Verify if the cbt module is enabled: esxcli system module list --enabled=true | grep cbt
• Verify if the cbt module is loaded : esxcli system module list --loaded=true | grep cbt
• To Manually load the module: # vmkload_mod cbt

The above changes aren't persistent across boot. CBT Module load issues is also reported to be in relation to known boot bank issue. To address the issue, please follow the steps mentioned in VMware knowledge base article https://knowledge.broadcom.com/external/article?legacyId=2149444