VMware Response to RSBA Mitigation Performance Questions
Article ID: 330027
Updated On:
Products
Issue/Introduction
The purpose of this article is to answer recuring questions and provide additional details with regard to the performance impact of mitigating RSBA-related vulnerabilities.
Q: Why are some VMs slower after upgrading to Linux kernel 5.19?
A: The Linux kernel maintainers, following advice from Intel, for CPUs with Return Stack Buffer Alternative (RSBA), changed default CPU vulnerability mitigation to Indirect Branch Restrict Speculation (IBRS). IBRS has a higher performance cost than previous default CPU vulnerability mitigations.
Q: Did VMware's July patches to vSphere cause this performance loss?
A: No, VMware's July patches mitigated RSBA-related vulnerability at the host level (preventing malicious VMs from attacking vSphere or other VMs), and with no measurable performance impact. VMware recommends installation of the July patches. The patches did not change VMware's optimization of virtualization of guest CPU mitigations. This implementation was created and optimized to minimize hypervisor overhead in 2018.
Q: Can I reconfigure my hosts or cluster to avoid performance regressions while running with Linux default CPU mitigations?
A: A VM on a newer host CPU (see list below) not enumerating RSBA can run with significantly reduced mitigation performance costs. The host must either:
- not be in an Enhanced vMotion Compatibility (EVC) cluster, or
- be in an EVC cluster of mode Cascade Lake or Ice Lake.
Q: Does VMware issue specific recommendations for in-guest mitigation of RSBA?
A: No. VMware enables a customer to implement the security policy if their choice, including in-guest CPU vulnerability mitigation configuration. This choice is up to the customer. Different mitigations will have different performance and/or security properties. Consult with the OS vendor for further details.
Resolution
What is RSBA?
Return Stack Buffer Alternative (RSBA) is a behavior of some Intel CPUs of the Skylake family. RSBA potentially enables an information leakage via an attack on Return Stack Buffer Underflow (RSBU). For concerned parties running software on vulnerable CPUs, this attack can be mitigated. Mitigation incurs a performance penalty.
Why is RSBA noteworthy now?
In July 2022, researchers demonstrated exploitability of RSBU. Many vendors' existing mitigations of previous CPU vulnerabilities were insufficient to mitigate RSBU. Software vendors have added and/or are adding RSBU mitigation, potentially incurring performance penalties.
Mitigation of RSBA in vSphere
For VMware, mitigation of RSBA-based attacks fall into two categories:
- Hypervisor-Specific Mitigation
- Guest Mitigation
Hypervisor-Specific Mitigation
Mitigates leakage from the hypervisor or guest VMs into a malicious guest VM. In July 2022, VMware released patches to vSphere, documented in VMSA-2022-0020, implementing hypervisor-specific mitigation at no visible performance cost.
Guest Mitigation
Mitigates leakage between processes within the VM, or between the VM's kernel and user processes. In 2018, VMware implemented hypervisor-assistance of guest mitigation by virtualizing speculative execution control mechanisms. VMware optimized this assistance to minimize hypervisor overhead. Within a VM, a guest operating systems may implement whatever mitigation it chooses (or none at all) and vSphere will execute this faithfully.
Performance Concerns and Guest Mitigation
Linux kernel 5.19 implements a new default mitigation for CPU vulnerabilities. If RSBA is detected in the underlying CPU, RSBU is mitigated using IBRS. IBRS is more expensive than the previous default mitigation (retpoline), resulting in performance loss. This performance loss is specific to workload and underlying physical CPU. Performance loss is experienced whether the underlying platform is physical (bare metal) or virtual (running on a hypervisor such as vSphere).
Windows already mitigated existing CPU vulnerabilities using IBRS by default, so there is no new overhead for RSBU mitigation in Windows guests.
Please refer to the VMware Performance Blog for additional information.
RSBA Performance Considerations on vSphere
RSBA is advertised to a guest VM unless either of these following:
- The host CPU does not enumerate RSBA (see above table of invulnerable CPUs) and the host CPU is not in a cluster in a Skylake or earlier EVC mode.
- The VM is configured with virtual hardware version 8 or earlier (in which case, the VM will receive no CPU mitigation virtualization at all as well as none of the benefits of newer virtual hardware). This is typically true of older, legacy VMs and virtual appliances.
VMware's patches in 2018 implemented this RSBA behavior. This was not changed by VMware's July patches.
Options for Regaining Performance
VMware enables customers to implement their choice of security policies, within their virtual machines. VMware enables these mitigation options with minimal hypervisor overheads. Customers may choose whether and how to mitigate given their policies. Operating system providers may provide mitigation alternatives with different security and performance properties.
A heterogeneous cluster of hosts (vulnerable and invulnerable) with a Skylake or older EVC mode, will advertise RSBA to all VMs therein. Splitting this cluster into different clusters (one vulnerable, one invulnerable) would allow the invulnerable cluster's VMs to run without RSBA advertised, recouping performance for those VMs.
Why is AMD not relevant to this discussion?
AMD CPUs are affected by a similar vulnerability called Branch Type Confusion (BTC). Default Linux mitigation of BTC on AMD uses different techniques than the above, with a different performance footprint. These defaults were added to the Linux kernel before version 5.19. As AMD does not have RSBA/RSBU but instead BTC, the kernel affecting the change was different and the performance effects are different, this is omitted from broader discussion.
Feedback
