Linux Guest OS transitions to read-only state during rapid snapshot operations with Dell RecoverPoint VAIO filter (emcsplitter)
search cancel

Linux Guest OS transitions to read-only state during rapid snapshot operations with Dell RecoverPoint VAIO filter (emcsplitter)

book

Article ID: 437252

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

 After a successful Storage vMotion between VVol datastores, the source VMDK file is not deleted, leading to 'vmdk already exists' errors on subsequent migration attempts.

Environment

ESXi 7.x and above

 Pure Storage Purity//FA 6.2.8+ or 6.3.1+
 VASA Provider 2.0.0

Cause

 ESXi host aborts stalled SCSI commands (DID_ABORT) directed at the affected virtual machine.

 The `vmware.log` of the affected virtual machine records back-to-back snapshot operations (consolidation followed immediately by creation) within seconds.

 The `vmware.log` records critical internal memory errors from the RecoverPoint filter (`emcsplitter`) between snapshot stun cycles:

  `RPS:#0 - CommandSplitterReportState_create: failed to create command. static memory already in use for 0 seconds`

The following sequence from `vmware.log` validates the fault domain:

1. Snapshot Consolidation Completes:

2026-03-13T08:38:33.921Z In(05) vcpu-0 - VigorTransport_ServerSendResponse opID=##### seq=10758: Completed Snapshot.Consolidate request in 8415832 US.
2026-03-13T08:38:33.933Z In(05) vcpu-0 - ConsolidateEnd: Snapshot consolidate complete: The operation completed successfully (0).

2. VAIO Filter Memory Error  

2026-03-13T08:38:38.797Z In(05) worker-65825933 - [65825917/2917578496]: RPS:#0 - CommandSplitterSetState_v_execute: failed to set the volume state, guid is: 0xefaec557a386f086
2026-03-13T08:38:38.797Z In(05) worker-65825933 - [65825917/2917578496]: RPS:#0 - CommandSplitterReportState_create: failed to create command. static memory already in use for 0 seconds

3. Subsequent Snapshot Request:

2026-03-13T08:38:49.409Z In(05) vmx - VigorTransportProcessClientPayload: opID=##### seq=11134: Receiving Snapshot.Take request.
2026-03-13T08:38:49.410Z In(06) vmx - Vigor_ClientRequestCb: Dispatching Vigor command 'Snapshot.Take'
2026-03-13T08:38:49.410Z In(05) vmx - SnapshotVMX_TakeSnapshot start: 'VM Snapshot 3/13/2026, 10:38:41 AM'...

4. Filter Failure During Stun:

2026-03-13T08:38:49.420Z Er(02) vmx - DISKLIB-LIB_CREATE   : DiskLib_NotifySnapshotPrepare: FiltLib failed to notify snapshot prepare: Operation completes asynchronously
2026-03-13T08:38:49.551Z In(06) Upcall-e4a83391 - FiltLib: Received stun notification
2026-03-13T08:38:49.551Z In(06) Upcall-e4a83391 - FiltLib: FiltLib_DiskStun : Stunning all IO filters.
2026-03-13T08:38:49.551Z In(05) Upcall-e4a83391 - [65825917/2950817536]: SplDiskStun: called with handle 486ACB8CF0

Resolution

The underlying fault resides within the third-party Dell RecoverPoint VAIO filter (VIB: `EMC-RP4VMs-SPL`). 

Contact Dell RecoverPoint Support to report the 'emcsplitter' static memory allocation failure during rapid snapshot operations. They are aware of the issue and can assist. 

 

Additional Information

Workaround:
To immediately mitigate the issue and prevent production outages, implement one of the following workarounds:

Workaround 1: Modify Backup/Snapshot Schedules
Review the backup schedules for critical Linux VMs. Space out the backup jobs or snapshot creations to ensure the virtual machine is not subjected to a new snapshot request immediately following a consolidation event. 

Workaround 2: VAIO Filter Bypass
If modifying the backup cadence is not feasible, temporarily remove the Dell RecoverPoint storage policy from the affected virtual machines. This action detaches the filter from the VM's vDisk data path, bypassing the `emcsplitter` daemon and preventing the I/O stalls.