Interacting with VMware Support Log bundles within ESXi may impact vSAN Object Accessibility
Article ID: 326531
Updated On:
Products
VMware vSAN
Issue/Introduction
Symptoms:
After unzipping a VMware Support bundle within an ESXi Host or attached storage, and then having removed the log bundle via a User Interface (UI) or Application Programming Interface (API) call (such as through the vCenter Datastore Browser or PowerShell), VMs may become inaccessible.
After unzipping a VMware Support bundle within an ESXi Host or attached storage, and then having removed the log bundle via a User Interface (UI) or Application Programming Interface (API) call (such as through the vCenter Datastore Browser or PowerShell), VMs may become inaccessible.
Cause
Each VMware Virtual Disk (VMDK) consists of two parts: a “VMDK descriptor File”, and its binary container where all user/VM-data is stored (historically a -flat.vmdk file, or on vSAN a vSAN ‘object’).
VMware vSAN Virtual Machine Disks (VMDKs) are abstracted so that they can function just like any other “file” on any other datastore, even though they are “object-based” storage objects so that a user can interact with them using a User Interface (UI) or API tools located within VMware and/or 3rd party software solutions.
This abstraction requires back-end operational logic so that when a virtual disk descriptor file (VMDK file), or what is visible to a UI like the Datastore Browser or PowerShell, is interacted with (created, moved, deleted) the backing vSAN object is also changed.
During vm-support creation of a log bundle, all VMDK descriptor files for registered VMs running on the Host where logs are being gathered are copied into the log bundle as they may be needed for Support analysis.
Note: No customer data is copied into the log bundle, only the “descriptor” files.
When a log bundle from an ESXi Host is unzipped within an ESXi Host, it introduces “copies” of these VMDK descriptor files within the unzipped location. By interacting with these VMDK descriptor "copies" through the vSphere Datastore Browser UI or vSphere API, the "copies" are indistinguishable from the real/original files currently in use by the VMs, and as a result, if a delete is issued against these "copies" of an unzipped log bundle, vSphere may process these delete requests to delete the live VM vSAN backing objects.
VMware vSAN Virtual Machine Disks (VMDKs) are abstracted so that they can function just like any other “file” on any other datastore, even though they are “object-based” storage objects so that a user can interact with them using a User Interface (UI) or API tools located within VMware and/or 3rd party software solutions.
This abstraction requires back-end operational logic so that when a virtual disk descriptor file (VMDK file), or what is visible to a UI like the Datastore Browser or PowerShell, is interacted with (created, moved, deleted) the backing vSAN object is also changed.
During vm-support creation of a log bundle, all VMDK descriptor files for registered VMs running on the Host where logs are being gathered are copied into the log bundle as they may be needed for Support analysis.
Note: No customer data is copied into the log bundle, only the “descriptor” files.
When a log bundle from an ESXi Host is unzipped within an ESXi Host, it introduces “copies” of these VMDK descriptor files within the unzipped location. By interacting with these VMDK descriptor "copies" through the vSphere Datastore Browser UI or vSphere API, the "copies" are indistinguishable from the real/original files currently in use by the VMs, and as a result, if a delete is issued against these "copies" of an unzipped log bundle, vSphere may process these delete requests to delete the live VM vSAN backing objects.
Resolution
VMware has mitigated this behavior in VMware vSAN 8.0 and above by limiting this back-end logic to the vSAN datastore. VMware recommends all VMware vSAN customers upgrade to the latest release of VMware vSAN 8.x to avoid this issue.
Note: Please do not interact with any log bundles within the VMware vSAN Datastore!
When collecting ESXi support bundles that use encryption be sure to follow Collect a vm-support Package for an ESXi Host That Uses Encryption and provide the password for the core dump.
Workaround:
VMware recommends refraining from interacting with Support Log Bundles within ESXi, please export and download them through the UI and interact with them on other appropriate platforms. You may also utilize tools like VMware Skyline or Skyline Health Diagnostics to assist you in log analytics rather than pursue investigations yourself.
If support requests you to decrypt the core dump as the password was not provided via Decrypt or Re-Encrypt an Encrypted Core Dump and running vSAN below 8.0, when it comes time to delete the unzipped bundle delete the files via ESXi command line by running rm -R <name of support bundle>
Note: Please do not interact with any log bundles within the VMware vSAN Datastore!
When collecting ESXi support bundles that use encryption be sure to follow Collect a vm-support Package for an ESXi Host That Uses Encryption and provide the password for the core dump.
Workaround:
VMware recommends refraining from interacting with Support Log Bundles within ESXi, please export and download them through the UI and interact with them on other appropriate platforms. You may also utilize tools like VMware Skyline or Skyline Health Diagnostics to assist you in log analytics rather than pursue investigations yourself.
If support requests you to decrypt the core dump as the password was not provided via Decrypt or Re-Encrypt an Encrypted Core Dump and running vSAN below 8.0, when it comes time to delete the unzipped bundle delete the files via ESXi command line by running rm -R <name of support bundle>
Feedback
Yes
No
Powered by
