Drives within the GuestOS of a VM become unavailable or missing after a VMware Tools upgrade

Products

• VMware vSphere ESXi

Issue/Introduction

• After upgrading VMware Tools, drives within the GuestOS of the VM become unavailable or missing resulting in the drives showing as offline. 

Environment

• VMware vSphere 8.x
• VMware ESXi 8.x

Cause

• When upgrading VMware Tools, this can cause disks to go offline within the Guest OS due to changes in the storage and network device drivers it provides. Primarily by changing the device naming or pathing.

Driver Changes and Device Naming

• Driver Replacement:
  • A VMware Tools upgrade often replaces existing storage drivers (like PVSCSI or LSI Logic) and network drivers (like VMXNET3) with newer versions.
• Device Path Alterations:
  • The new drivers might slightly alter how the Guest OS identifies and enumerates the storage devices. For instance, a disk previously identified as /dev/sdb (on Linux) or Disk 1 (on Windows) might be seen as /dev/sdc or Disk 2 after the driver change.
• Persistent Naming Conflict:
  • The Guest OS relies on persistent naming rules or configuration files (like /etc/fstab on Linux or the Disk Management console in Windows) to mount or bring specific disks online. These configurations are often tied to the previous device path or identifier. If the device path changes, the OS cannot locate the disk using its old configuration, causing it to remain offline.

Specific Scenarios

• Linux Guests:
  • The /etc/fstab file often uses UUIDs (Universally Unique Identifiers) or specific device names (/dev/sdX) to mount filesystems at boot. While UUIDs are generally persistent, certain edge cases or misconfigurations might cause issues. If the fstab uses device names (/dev/sdX), the upgrade is more likely to cause issues due to device re-enumeration.
• Windows Guests:
  • Windows manages disk enumeration and drive letters in the Disk Management console and the registry. Driver changes can cause Windows to see the "new" disks as foreign or unconfigured until they are manually re-assigned or brought online.
• Multipath Configurations: 
  • Environments using multipathing software to handle redundant connections to storage are particularly vulnerable. The upgrade might change the underlying physical paths (e.g., sda to sdb), breaking the multipath configuration until the new paths are manually re-configured or scanned by the multipath software. 

Resolution

Typically, these issues are resolved by:

• Reconfiguring the Guest OS: Manually editing configuration files (like /etc/fstab) or using OS-specific tools to identify the new device paths or UUIDs and update the settings.
• Navigating into the Disk Management utility within the GuestOS and selecting the disk(s) labeled offline and selecting online.
• If the drives appear missing, within the disk management utility you can perform a rescan operation to make the drives reappear by performing the following:
  • Open Disk Management (right-click Start in Windows or use diskmgmt.msc).
  • Right-click on Disk Management in the left pane and select Rescan disks. This often makes the drives appear without a full reboot.