SD card/USB boot device revised guidance

Products

• VMware vSphere ESXi

Issue/Introduction

Background


vSphere customers have encountered many issues related to reliability and performance when SD cards/USB media are used as standalone boot devices. Endurance ratings alone cannot guarantee the viability of SD cards/USB media as a boot option. Mitigations and appropriate messaging through KB articles are already in place to help current customers on vSphere 7.x. An ESXi boot device consists of several partitions. One of the boot partitions, the ESX-OSDATA partition is important for ESX operations. The OSData contains the VMTools and scratch areas that are critical regions.

The following diagram illustrates the transition of boot partition layouts used on the boot devices across the different vSphere releases:




Summary
VMware will continue supporting USB/SD card as a boot device through the vSphere 8.0 product release, including the update releases. Both installs and upgrades will be supported on USB/SD cards. The change from the previous guidance is that SD/USB as a standalone device will now be supported on previously certified server platforms. Customers can still however provision an additional persistent device to store the OSData partition, which VMware recommends. ESX will attempt to relocate the critical regions to such a persistent device. Starting from 7.0U3c, if an USB SD card boot device is detected during installs and upgrades, critical regions of the OSData partition such as VMTools and scratch will automatically be moved out. Moreover with vSphere 8.0, the entire OSData partition can be moved out.

The upgrade or install workflows for vSphere 8.0 will ensure that the OSData partition is relocated away from USB/SD card into a persistent device. There will be an automatic fallback to use a VMFS datastore, or a RAMDisk if such a device is not available. Preferably, the SD cards should be replaced with an SSD or another local persistent device as the standalone boot option.

VMware is working very closely with all the major OEMs to ensure that future generations of server platforms  do not support USB/SD card as a boot device and that more reliable mechanisms that conform to the VMware device and endurance requirements below for boot devices are provided on servers.
Caution:
For any new hardware procurement, VMware recommends that customers evaluate available options carefully, and plan to use a persistent device for boot that meets ESX endurance requirements as specified in the Guidelines for vSphere Boot Devices section below. While ordering new hardware for their environment, customers should ensure that SD cards/USB media are not chosen for the boot device. Newer OEM servers/systems that carry SD/USB as a boot device will not certify successfully.

Potential issues seen with SD card/USB as a boot option

vSphere customers have encountered many issues related to device reliability when SD cards are used as the boot device. Endurance ratings can no longer guarantee the viability of SD cards as a boot option.
 
Some of the reasons why SD cards will be unsustainable with the next major release are:

1. Reads/Writes to System Storage continue to grow, to the ESX-OSDATA partition. The OSDATA partition is a critical component that is required to be always available and be persistent across reboots. OSDATA will be used in many ways as services and applications such as vSAN and NSX grow. OSDATA is used to store some of the runtime state, and for probes, backups of items such as configuration, timestamps etc. Access to other areas on OSDATA, such as VMTools and scratch portions will also grow. With each new vSphere release, more features will start relying on the OSDATA partition.
2. Performance demands and IO loads cannot be continued to be met by SD cards. For example, the demands on writes to the scratch area have been growing. Another concern is the high frequency of IO with bursts such as logs or traces.
3. There is no way to reliably check and monitor for the SD card endurance, as these devices usually do not provide support for diagnostics data. Wear-out issues and remaining endurance or lifetime can go undetected. In addition, not just writes but reads are also known to cause wear issues.
4. SD/USB devices are prone to wear over time and are thus subject to failures, they are not designed for enterprise class use-cases.

Please note that mitigations and appropriate messaging through KB articles are already in place to help current customers to run reliably on 7.x. Please refer to section below on Options for 7.x where we provide workarounds.

Resolution

Options for vSphere 8.0
 
Customers will not see any changes for their upgrade workflows. Customers can continue using their existing boot devices, even if it is a USB/SD card boot device, and OSData will be safely relocated to the best available location.
For upgrades and installs, customers now have an option of selecting a persistent device to store the OSData partition.

One of the following options will be used to store the OSData partition:

1. Local Persistent Disk

Use the entire local persistent disk as the boot device, including boot bank and OSDATA partitions. Plan to use a local persistent disk as the boot device for all partitions.

1. • If a local persistent device is not already available, add another persistent device. This persistent local boot device can be industrial grade SAS, SATA or PCIe NVMe disk (32GB** minimum, 128GB** recommended)
2. SD Card/USB Media + Local Persistent Disk

If a persistent local device is not available as a boot device, SD cards can be used for boot bank partitions However a separate persistent local device to store the OSDATA partition (32GB** minimum, 128GB** recommended) must be provided

3. SAN/NAS LUN for the entire system storage

Provide SAN/NAS Boot as an option for the entire System Storage (Boot banks + OSDATA partitions), when ESXi is installed on a SAN/NAS LUN. In case where a LUN is configured to store just the OSData partition, a new VMFS-L partition will be created that will not be shareable.

4. VMFS partitions

Installs/upgrades to vSphere 8.0 will also provide options to store OSData into an existing VMFS partition. For customers who have existing server installations of SD cards/USB media for boot and are trying to upgrade to vSphere 8.0, an option to use a VMFS datastore will also be provided, into which customers will be able to configure and store the OSDATA partition. A separate unique directory will be created to store OSData for each host that uses the same VMFS datastore.

5. RAMDisk

The RAMDisk option will automatically be a final resort and will be used if none of the above options are available

Options for 7.x
 
It is strongly recommended that customers start planning to move away completely from SD card usage by following the options listed above for future releases.
 
However, it is important to note that we are providing exceptions on SD card usage with 7.x and will continue to provide  support.
SD cards, even though not an ideal option, can still be used as a standalone boot option to store all partitions including OSDATA, through all vSphere 7.x releases. 

Customers on previous versions (for example, 6.x) of vSphere should feel confident to move to 7.0U2c as several critical issues have been fixed with this release, and some workarounds have been provided as default for the VMTools & Scratch areas.
Note that RAMDisk as storage for OSDATA is also an option that is supported through 7.x and vSphere 8.0 releases.


For fixes for critical storage stack issue available in 7.0 Update 2c
  See:
Bootbank cannot be found at path '/bootbank' errors being seen after upgrading to ESXi 7.0 U2 (317914)

VMware ESXi 7.0 Update 2c Release Notes
  (Please look for USB device and Storage stack issues fixed with this release)

Customer actions for 7.x
Important for 7.x: Customers must ensure that the following fixes and workarounds are in place.

1. Customers must move to 7.0U2C or above. Critical fixes, as outlined in the release notes link above, are available with vSphere 7.0 Update 2c, 7.0 update 3 and above releases
2. For customers planning to upgrade to the next release, ESX will automatically move VMTools to RAMDisk. See Connection to the /bootbank partition intermittently breaks when you use USB or SD devices (318611)
3. For VMware customers on vSphere 7.0 update 2c, there are options to manually configure VMTools to move RAMDisk. See ToolsRamdisk option is not available with ESXi 7.0.x releases (313868)
4. For customers planning to run 7.X releases, some of the workarounds customers must apply are given below:
   1. Reduce IOs to SD Card by reducing access to VMTools and scratch areas of OSDAT.

   2. For VMTools area of OSDATA, by default, ESX will try to find a local persistent device or a SAN LUN to store VMTools. Customers can also enable options to move VMTools out of SD card into RAMDisk.
       See Connection to the /bootbank partition intermittently breaks when you use USB or SD devices(318611) and High frequency of read operations on VMware Tools image may cause SD card corruption (318794)

   3. For Scratch area, if there is no local HDD, SSD, NVMe device, or M.2 found, ESX will move the scratch area as well to RAMDisk automatically. 
   4. Customers can also configure and move Scratch area to SAN or NFS. See: Creating a persistent scratch location for ESXi 8.x/7.x/6.x (317689)
   5. In addition, customers also have options to configure remote syslog to move logs out of OSDATA. See: Configuring syslog on ESXi (318939)

5. Customers can expect to see warnings with vSphere 7.0U3 shown in Persistent storage warnings when booting ESXi from SD-Card/USB devices. (317891)

Guidelines for vSphere Boot Devices

A SATA/SAS/NVMe SSD that meets VMware specified endurance requirements, or HDD is strongly recommended.

Boot Device:
High-Quality Media: PCIe NVMe, SAS, SATA/SATADOM SSD
Industrial grade flash/SSDs, SLC or pSLC

Boot Device Size:
32GB** minimum (must), 128GB** recommended

Endurance requirements:
128 TBW (over 5 years)

IO Performance
At least 100MB/s
Please refer to the ESXi hardware requirements here:  ESXi Hardware Requirements

Supported=Yes, Not Supported=No	7.0U2c & 7.0U3	vSphere 8.0
SD card as boot device	Yes, Not recommended	Yes, Not recommended
Use local Persistent disk for ESX-OSDATA	No	Yes
Move VMTools area away from SD card/USB (out of ESX-OSDATA)	Yes, Recommended	No (Not required since OSDATA itself will be moved out)
Move Scratch area away from SD card/USB (out of ESX-OSDATA)	Yes, Recommended	No (Not required since OSDATA itself will be moved out)
Move ESX-OSDATA out of SD card	No	Yes
RAMDisk as storage for boot areas like VMTools & scratch	Yes, Not Recommended	Yes, Not Recommended
RAMDisk as storage for ESX-OSDATA partition	No	Yes, Not recommended
ESX-OSDATA partition can be carved out of existing VMFS datastore (share space)	No	Yes

FAQ 
Based on the boot device used, could an install or upgrade to 7.x or vSphere 8.0 be blocked?
VMware will not block ESXi install or upgrade. It will, however, try to relocate critical regions of the boot device into persistent storage.

What is VMware telling its partners?
VMware is working very closely with its OEM partners. All major OEM server vendors today support options such as an M.2, BOSS, NVMe, SATA with a high-endurance boot device in their current generation of servers. In addition, OEM server vendors have agreed to stop supporting USB/SD card-based boot device with servers having the newer generation of CPU platforms such as Sapphire Rapids and Genoa. Any new server storage for boot devices must comply with vSphere 8.0 endurance requirements. VMware has also requested OEM partners to remove any option to purchase USB/SD as boot device, such as with web-based ordering tools, sales tools, and with channels or resellers.

Is there any certification impact?
Apart from already certified servers, VMware server certification process will block any new server certification for servers that have a USB/SD card as the boot device.
In addition, VMware has stopped certifying flash devices for boot device, and will stop listing them as approved devices starting in vSphere 8.0. Devices already certified for previous vSphere releases will continue to be supported in vSphere 8.0 and future releases until device EOL (End of Life)


Why should customers move now to a persistent boot device for any new servers?
There are two main reasons to recommend a move now rather than later:
1) To avoid managing 2 separate devices to store boot partitions - SD/USB and another device
2) Future-proofing customer infrastructure

• SD/USB options eventually removed
• New platforms will not be allowed to be certified for USB/SD cards
  • Sapphire Rapids/Genoa platforms not certified
• OEMs have been asked to take specific actions to not support USB/SD cards for their future platforms and in their sales process.

What is the impact of storing OSData on RAMDisk?
With vSphere 8.0, RAMDisk will be used as a last resort option to store OSData. Customers can ensure that if deemed important, traces and logs are configured to be stored separately. Similarly, coredumps can be configured to be stored on alternate locations.

Can application data or VMFS partitions be stored on USB/SD card boot devices?
VMware strongly recommends that if USB/SD card is used as the boot device, customers not store any application (VM) data intentionally on boot devices. Some use cases can lead to a lot of Reads/Writes, for example, in some instances, databases are stored on scratch, or new VMFS partitions are created on USB/SD card. Such use of the USB/SD card boot device should be avoided.

How is the VMTools location stored and controlled?
VMTools region is a part of OSData and will be moved to any place where OSData is stored. However, it can be controlled by the ProductLocker advanced setting. At install/upgrade time, it is in the boot device VMFS-L partition (either the LOCKER partition on USB/SDCard, or the OSDATA partition if the OSDATA is installed to non-USB). When the system boots up, the system-storage jumpstart will check if it is in LOCKER on USB/SD device, and if there is a local OSDATA partition, it will be moved there. The customer can change the location anywhere else by setting the ProductLocker value. (Also see Installing and upgrading the latest version of VMware Tools on existing hosts (313876))

What type of devices and interfaces is this KB referring to?
We are referring to devices that are unreliable, such as SD cards with unpredictable and unreportable endurance.
We are also referring to unreliable interfaces such as USB.
Please note that when we refer to SAS, SATA (including SATADOM), NVMe, etc., we imply that these devices must be supported in their native formats and not through any conversions (such as NVMe or SAS to USB)

Can OSData be pointed to use a VMFS partition?
VMware is now allowing for such use. On an install/upgrade to vSphere 8.0, OSData can be stored on a VMFS volume with sufficient space (minimum 32GB**)

Can OSData be pointed to use a remote disk?
VMware cautions against configuring and using OSData on a shared remote LUN during install/upgrade. The management of such a use-case is prone to risk as there is no way to control how such LUNs will be coordinated for access especially when multiple hosts can be configured to use the same LUN.

Can the boot LUN be expanded?
Technically, expansion of the LUN can only work if there is no other partition after the VMFS-L OSDATA partition. Usually there is a datastore partition there when upgrading from 6.x to 7.0, and that prevents stretching OSData when there is a larger LUN. If upgrading from 6.x to 7.0:

• If there is a datastore on the LUN, then no expansion can happen on upgrade
• If the LUN is increased in size while in 6.x, followed by upgrading and reboot, then when 7.0 comes up it will repartition the boot device such that system-storage partitions and will occupy the space (up to 128GB**) of the LUN => this is the only case where expansion can happen

If ESXI is upgraded to 7.0, then resize the LUN => the system is in upgraded state and no partition expansion will occur.


For more FAQ, see ESXi System Storage FAQ

** GB here equals 1024x1024x1024 (2^30=1,073,741,824) bytes as opposed to 10^9 bytes (1,000,000,000)

 

Additional Information

Please refer to the following blog:
vSphere 7 – ESXi System Storage Changes