Linux VMs with PVSCSI controllers experience read-only filesystems during snapshot operations when using Dell RecoverPoint VAIO

Products

• VMware vSAN

Issue/Introduction

 

• Linux virtual machines become intermittently unresponsive or inaccessible.

• The guest OS filesystem (e.g., EXT4) unexpectedly remounts as read-only.

• Guest OS kernel logs (dmesg or /var/log/messages) display block layer I/O errors and SCSI aborts matching the following signatures:

  • end_request: I/O error, dev sda, sector <N>

  • hostbyte=DID_ABORT driverbyte=DRIVER_OK

  • Buffer I/O error on device dm-0, logical block <N>

  • EXT4-fs ...: ext4_journal_start_sb: Detected aborted journal

  • EXT4-fs (dm-0): Remounting filesystem read-only

• This issue is observed during snapshot creation or consolidation (e.g., Commvault backup jobs).

• The affected VMs are configured with the VMware Paravirtual (PVSCSI) storage controller.

• The affected VMs are protected by Dell RecoverPoint for VMs and have the VAIO I/O filter (emcsplitter) actively attached in their storage polic

 

Environment

 

• VMware vSphere ESXi 8.0.x (and potentially earlier versions)

• Dell RecoverPoint for VMs (using VAIO emcsplitter VIB, e.g., EMC-RP4VMs-SPL)

• Guest OS configured with VMware Paravirtual (PVSCSI) controllers

• Affected Guest Operating Systems:

  • Ubuntu 14.04, 16.04, 18.04, 20.04 (partially)

  • Oracle Linux 6.10

  • Any other OS running Linux kernel 4.15 or older

 

Cause

 When virtual machines protected by the Dell RecoverPoint VAIO filter undergo rapid snapshot operations (stun/unstun cycles), I/O requests are temporarily delayed or blocked at the hypervisor level while the filter manages its internal buffers and state. If these delays exceed thresholds, the ESXi storage stack issues a hard SCSI abort (DID_ABORT) to prevent the host from locking up.

Older versions of the Linux pvscsi driver do not gracefully handle these specific aborts and delays during the VAIO stun window. Consequently, the I/O failure is passed up to the guest OS filesystem. To protect data integrity, EXT4 intentionally aborts its journal and remounts as read-only.

While a significant related upstream fix for the pvscsi driver was introduced in Linux kernel 4.14.51, testing indicates that distributions utilizing slightly newer base kernels (such as Ubuntu 18.04 using 4.15) may still lack all necessary backports to fully mitigate this specific VAIO interaction. 

 

Resolution

Because the pvscsi driver is maintained within the Linux kernel, there is no patch available from the hypervisor or VMware Tools to resolve the driver's behavior.

To permanently resolve the issue, perform one of the following:

• Upgrade the Guest OS: Upgrade the affected Linux virtual machines to a modern distribution/kernel version that includes the resilient pvscsi driver updates (e.g., Ubuntu 22.04 or later, RHEL 8.6 or later).

• Request an OS Vendor Backport: If an OS upgrade is not feasible, contact your Linux distribution vendor to request the latest pvscsi driver fixes be backported to your current kernel version.

Additional Information

If you cannot upgrade the Linux operating system, you can mitigate the issue using one of the following workarounds:

Workaround 1: Change the Virtual SCSI Controller Testing has confirmed that the LSI Logic controller gracefully handles the snapshot stuns.

1. Power down the affected virtual machine.

2. Edit the Virtual Machine settings.

3. Expand the SCSI controller and change the type from VMware Paravirtual to LSI Logic Parallel or LSI Logic SAS.

4. Power on the virtual machine. Note: Consult Broadcom documentation regarding performance considerations when moving high-I/O workloads away from PVSCSI.

Workaround 2: Space Out Backup Schedules The issue is easily triggered by back-to-back snapshot operations. Adjust your backup software (e.g., Commvault) schedules to ensure random or extended delays between snapshot creations and consolidations, preventing overlapping stun states.

Workaround 3: Temporarily Bypass the VAIO Filter For critical systems where controller changes or schedule adjustments are not immediately possible:

1. Edit the Virtual Machine settings.

2. Change the VM Storage Policy to a default vSAN or VMFS policy that does not include the Dell RecoverPoint VAIO components.

3. This detaches the emcsplitter, allowing I/O to flow directly to the storage backend without interception.