Background
vSphere customers have encountered many issues related to reliability and performance when SD cards/USB media are used as standalone boot devices. Endurance ratings alone cannot guarantee the viability of SD cards/USB media as a boot option. Mitigations and appropriate messaging through KB articles are already in place to help current customers on vSphere 7.x. An ESXi boot device consists of several partitions. One of the boot partitions, the ESX-OSDATA partition is important for ESX operations. The OSData contains the VMTools and scratch areas that are critical regions.
Please refer to the following blog:
vSphere 7 – ESXi System Storage Changes
The following diagram illustrates the transition of boot partition layouts used on the boot devices across the different vSphere releases:
Summary
VMware will continue supporting USB/SD card as a boot device through the vSphere 8.0 product release, including the update releases. Both installs and upgrades will be supported on USB/SD cards. The change from the previous guidance is that SD/USB as a standalone device will now be supported on previously certified server platforms. Customers can still however provision an additional persistent device to store the OSData partition, which VMware recommends. ESX will attempt to relocate the critical regions to such a persistent device. Starting from 7.0U3c, if an USB SD card boot device is detected during installs and upgrades, critical regions of the OSData partition such as VMTools and scratch will automatically be moved out. Moreover with vSphere 8.0, the entire OSData partition can be moved out.
The upgrade or install workflows for vSphere 8.0 will ensure that the OSData partition is relocated away from USB/SD card into a persistent device. There will be an automatic fallback to use a VMFS datastore, or a RAMDisk if such a device is not available. Preferably, the SD cards should be replaced with an SSD or another local persistent device as the standalone boot option.
VMware is working very closely with all the major OEMs to ensure that future generations of server platforms do not support USB/SD card as a boot device and that more reliable mechanisms that conform to the VMware device and endurance requirements below for boot devices are provided on servers.
Caution:
For any new hardware procurement, VMware recommends that customers evaluate available options carefully, and plan to use a persistent device for boot that meets ESX endurance requirements as specified in the Guidelines for vSphere Boot Devices section below. While ordering new hardware for their environment, customers should ensure that SD cards/USB media are not chosen for the boot device. Newer OEM servers/systems that carry SD/USB as a boot device will not certify successfully.
Potential issues seen with SD card/USB as a boot option
vSphere customers have encountered many issues related to device reliability when SD cards are used as the boot device. Endurance ratings can no longer guarantee the viability of SD cards as a boot option.
Some of the reasons why SD cards will be unsustainable with the next major release are:
Please note that mitigations and appropriate messaging through KB articles are already in place to help current customers to run reliably on 7.x. Please refer to section below on Options for 7.x where we provide workarounds.
Options for vSphere 8.0
Customers will not see any changes for their upgrade workflows. Customers can continue using their existing boot devices, even if it is a USB/SD card boot device, and OSData will be safely relocated to the best available location.
For upgrades and installs, customers now have an option of selecting a persistent device to store the OSData partition.
One of the following options will be used to store the OSData partition:
Use the entire local persistent disk as the boot device, including boot bank and OSDATA partitions. Plan to use a local persistent disk as the boot device for all partitions.
If a persistent local device is not available as a boot device, SD cards can be used for boot bank partitions However a separate persistent local device to store the OSDATA partition (32GB** minimum, 128GB** recommended) must be provided
Provide SAN/NAS Boot as an option for the entire System Storage (Boot banks + OSDATA partitions), when ESXi is installed on a SAN/NAS LUN. In case where a LUN is configured to store just the OSData partition, a new VMFS-L partition will be created that will not be shareable.
Installs/upgrades to vSphere 8.0 will also provide options to store OSData into an existing VMFS partition. For customers who have existing server installations of SD cards/USB media for boot and are trying to upgrade to vSphere 8.0, an option to use a VMFS datastore will also be provided, into which customers will be able to configure and store the OSDATA partition. A separate unique directory will be created to store OSData for each host that uses the same VMFS datastore.
The RAMDisk option will automatically be a final resort and will be used if none of the above options are available
Options for 7.x
It is strongly recommended that customers start planning to move away completely from SD card usage by following the options listed above for future releases.
However, it is important to note that we are providing exceptions on SD card usage with 7.x and will continue to provide support.
SD cards, even though not an ideal option, can still be used as a standalone boot option to store all partitions including OSDATA, through all vSphere 7.x releases.
Customers on previous versions (for example, 6.x) of vSphere should feel confident to move to 7.0U2c as several critical issues have been fixed with this release, and some workarounds have been provided as default for the VMTools & Scratch areas.
Note that RAMDisk as storage for OSDATA is also an option that is supported through 7.x and vSphere 8.0 releases.
For fixes for critical storage stack issue available in 7.0 Update 2c
See:
Bootbank cannot be found at path '/bootbank' errors being seen after upgrading to ESXi 7.0 U2 (317914)
VMware ESXi 7.0 Update 2c Release Notes
(Please look for USB device and Storage stack issues fixed with this release)
Customer actions for 7.x
Important for 7.x: Customers must ensure that the following fixes and workarounds are in place.
For VMTools area of OSDATA, by default, ESX will try to find a local persistent device or a SAN LUN to store VMTools. Customers can also enable options to move VMTools out of SD card into RAMDisk.
See Connection to the /bootbank partition intermittently breaks when you use USB or SD devices(318611) and High frequency of read operations on VMware Tools image may cause SD card corruption (318794)
Guidelines for vSphere Boot Devices
A SATA/SAS/NVMe SSD that meets VMware specified endurance requirements, or HDD is strongly recommended.
Boot Device:
High-Quality Media: PCIe NVMe, SAS, SATA/SATADOM SSD
Industrial grade flash/SSDs, SLC or pSLC
Boot Device Size:
32GB** minimum (must), 128GB** recommended
Endurance requirements:
128 TBW (over 5 years)
IO Performance
At least 100MB/s
Please refer to the ESXi hardware requirements here: ESXi Hardware Requirements
Supported=Yes, Not Supported=No | 7.0U2c & 7.0U3 | vSphere 8.0 |
SD card as boot device | Yes, Not recommended | Yes, Not recommended |
Use local Persistent disk for ESX-OSDATA | No | Yes |
Move VMTools area away from SD card/USB (out of ESX-OSDATA) | Yes, Recommended | No (Not required since OSDATA itself will be moved out) |
Move Scratch area away from SD card/USB (out of ESX-OSDATA) | Yes, Recommended | No (Not required since OSDATA itself will be moved out) |
Move ESX-OSDATA out of SD card | No | Yes |
RAMDisk as storage for boot areas like VMTools & scratch | Yes, Not Recommended | Yes, Not Recommended |
RAMDisk as storage for ESX-OSDATA partition | No | Yes, Not recommended |
ESX-OSDATA partition can be carved out of existing VMFS datastore (share space) | No | Yes |
FAQ
Based on the boot device used, could an install or upgrade to 7.x or vSphere 8.0 be blocked?
VMware will not block ESXi install or upgrade. It will, however, try to relocate critical regions of the boot device into persistent storage.
What is VMware telling its partners?
VMware is working very closely with its OEM partners. All major OEM server vendors today support options such as an M.2, BOSS, NVMe, SATA with a high-endurance boot device in their current generation of servers. In addition, OEM server vendors have agreed to stop supporting USB/SD card-based boot device with servers having the newer generation of CPU platforms such as Sapphire Rapids and Genoa. Any new server storage for boot devices must comply with vSphere 8.0 endurance requirements. VMware has also requested OEM partners to remove any option to purchase USB/SD as boot device, such as with web-based ordering tools, sales tools, and with channels or resellers.
Is there any certification impact?
Apart from already certified servers, VMware server certification process will block any new server certification for servers that have a USB/SD card as the boot device.
In addition, VMware has stopped certifying flash devices for boot device, and will stop listing them as approved devices starting in vSphere 8.0. Devices already certified for previous vSphere releases will continue to be supported in vSphere 8.0 and future releases until device EOL (End of Life)
(See Approved Flash Devices)
Why should customers move now to a persistent boot device for any new servers?
There are two main reasons to recommend a move now rather than later:
1) To avoid managing 2 separate devices to store boot partitions - SD/USB and another device
2) Future-proofing customer infrastructure
What is the impact of storing OSData on RAMDisk?
With vSphere 8.0, RAMDisk will be used as a last resort option to store OSData. Customers can ensure that if deemed important, traces and logs are configured to be stored separately. Similarly, coredumps can be configured to be stored on alternate locations (see KB 314320)
Can application data or VMFS partitions be stored on USB/SD card boot devices?
VMware strongly recommends that if USB/SD card is used as the boot device, customers not store any application (VM) data intentionally on boot devices. Some use cases can lead to a lot of Reads/Writes, for example, in some instances, databases are stored on scratch, or new VMFS partitions are created on USB/SD card. Such use of the USB/SD card boot device should be avoided.
How is the VMTools location stored and controlled?
VMTools region is a part of OSData and will be moved to any place where OSData is stored. However, it can be controlled by the ProductLocker advanced setting. At install/upgrade time, it is in the boot device VMFS-L partition (either the LOCKER partition on USB/SDCard, or the OSDATA partition if the OSDATA is installed to non-USB). When the system boots up, the system-storage jumpstart will check if it is in LOCKER on USB/SD device, and if there is a local OSDATA partition, it will be moved there. The customer can change the location anywhere else by setting the ProductLocker value. (Also see Installing and upgrading the latest version of VMware Tools on existing hosts (313876))
What type of devices and interfaces is this KB referring to?
We are referring to devices that are unreliable, such as SD cards with unpredictable and unreportable endurance.
We are also referring to unreliable interfaces such as USB.
Please note that when we refer to SAS, SATA (including SATADOM), NVMe, etc., we imply that these devices must be supported in their native formats and not through any conversions (such as NVMe or SAS to USB)
Can OSData be pointed to use a VMFS partition?
VMware is now allowing for such use. On an install/upgrade to vSphere 8.0, OSData can be stored on a VMFS volume with sufficient space (minimum 32GB**)
Can OSData be pointed to use a remote disk?
VMware cautions against configuring and using OSData on a shared remote LUN during install/upgrade. The management of such a use-case is prone to risk as there is no way to control how such LUNs will be coordinated for access especially when multiple hosts can be configured to use the same LUN.
Can the boot LUN be expanded?
Technically, expansion of the LUN can only work if there is no other partition after the VMFS-L OSDATA partition. Usually there is a datastore partition there when upgrading from 6.x to 7.0, and that prevents stretching OSData when you have a larger LUN. If you are upgrading from 6.x to 7.0:
If you upgrade, reboot to 7.0, then resize the LUN => the system is in upgraded state and no partition expansion will occur.
For more FAQ, see ESXi System Storage FAQ
** GB here equals 1024x1024x1024 (2^30=1,073,741,824) bytes as opposed to 10^9 bytes (1,000,000,000)