Resolving file locking conflicts and unexpected reboots while adding shared disk to the secondary node in the ORACLE RAC cluster
search cancel

Resolving file locking conflicts and unexpected reboots while adding shared disk to the secondary node in the ORACLE RAC cluster

book

Article ID: 427636

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms

  • After adding the shared disk to the secondary vm in ORACLE RAC cluster, below error message is observed

    Unable to access file since it is locked filePath: /vmfs/volumes/########-########-####-#############/vm_name/vm_name_#-rdmp.vmdk lockMode: 3 mac: ########-########-####-############# Locking conflict for file "/vmfs/volumes/########-########-####-#############/vm_name/vm_name_#-rdmp.vmdk". VMFS on-disk lock owner is host with MAC address ########-########-####-############# and VMFS lock mode Ox3. File system specific implementation of loctl[file] failed File system specific implementation of OpenFile[file] failed File system specific implementation of OpenFile[file] failed File system specificimplementation of OpenFile[file] failed File system specific implementation of OpenFile[file] failed File system specific implementation of OpenFile[file] failed File system specific implementation of OpenFile[file] failed Failed to lock the file Cannot open the disk '/vmfs/volumes/########-########-####-#############/vm_name/vm_name_#-rdmp.vmdk' or one of the snapshot disks it depends on. Failed topower on scsi0:6. Failed to add disk scsi0:6.

  • Sometimes, immediately after the operation fails, the Guest OS may trigger a kernel panic or a "CPU has been disabled by the guest operating system" event and the node is abruptly rebooted.

  • The primary node remains stable, but the secondary node fails to initialize the shared storage path.

Environment

VMware ESX 8.x

VMware ESX 9.x

Cause

This issue is observed when the shared disk  is added to the secondary node and is configured incorrectly compared to the primary node. Specifically:

  1. The disk is attached to SCSI Controller 0 (typically reserved for non-shared OS disks) instead of a dedicated storage controller.

  2. The multi-writer attribute which allows multiple ESXi hosts to access the same VMDK/RDM simultaneously without exclusive locks is not enabled on the secondary VM.

Because the primary VM holds an exclusive lock on the file, the secondary VM’s attempt to mount it without the multi-writer flag causes a fatal locking error. In some cases, Guest OS will trigger CPU halt and reboot to prevent data corruption.

Cause Validation:

Analysis of the logs confirms the sequence of events leading to the crash.

  • vmware.log of the secondary vm shows that the shared disk was added via scsi0:x. Immediately following the attachment, the log reports an inability to lock the file.

    2026-01-22T22:33:56.514Z In(05) vmx mkiovdc4-2859715-auto-1pakk-h5:70######-##-##-##-#### HotAdd: Adding scsi-hardDisk with mode 'independent-persistent' to scsi0:x
    2026-01-22T22:33:56.514Z In(05) vmx mkiovdc4-2859715-auto-1pakk-h5:70######-##-##-##-#### DISK: OPEN scsi0:x '/vmfs/volumes/########-########-####-#############/vm_name/vm_name.vmdk' independent-persistent R[]
    2026-01-22T22:34:00.523Z In(05) vmx mkiovdc4-2859715-auto-1pakk-h5:70######-##-##-##-#### AIOGNRC: Failed to open '/vmfs/volumes/########-########-####-#############/vm_name/vm_name_#-rdmp.vmdk' : Failed to lock the file (40003) (0x42003).
    2026-01-22T22:34:00.523Z In(05) vmx mkiovdc4-2859715-auto-1pakk-h5:70######-##-##-##-#### AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/########-########-####-#############/vm_name/vm_name_#-rdmp.vmdk' locked (try 0)

    2026-01-22T22:34:25.188Z In(05) vcpu-0 - Vix: [vmxCommands.c:7175]: VMAutomation_HandleCLIHLTEvent. Do nothing.
    2026-01-22T22:34:25.188Z In(05) vcpu-0 - MsgHint: msg.monitorevent.halt
    2026-01-22T22:34:25.188Z In(05)+ vcpu-0 - The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

    The log entry "CPU has been disabled by the guest operating system" indicates that the Linux kernel executed CLI and HLT instructions, intentionally halting the CPU because it could not safely access the required storage

  • ESXi host hostd.log file reveals explicit lock conflict events were recorded. The logs identify that the lock was currently held by the ESXi host where the primary node  was residing.

    2026-01-22T22:34:22.085Z Db(167) Hostd[2103529]: --> Cannot open the disk '/vmfs/volumes/########-########-####-#############/vm_name/vm_name.vmdk' or one of the snapshot disks it depends on.
    2026-01-22T22:34:22.085Z Db(167) Hostd[2103529]: --> Failed to lock the file
    2026-01-22T22:34:22.085Z Db(167) Hostd[2103529]: --> File system specific implementation of OpenFile[file] failed
    2026-01-22T22:34:22.085Z Db(167) Hostd[2103529]: --> File system specific implementation of OpenFile[file] failed
    2026-01-22T22:34:22.085Z Db(167) Hostd[2103529]: --> File system specific implementation of OpenFile[file] failed
    2026-01-22T22:34:22.085Z Db(167) Hostd[2103529]: --> Locking conflict for file "/vmfs/volumes/########-########-####-#############/vm_name/vm_name-rdmp.vmdk". VMFS on-disk lock owner is host with MAC address 6#######-########-####-############ and VMFS lock mode 0x3.

    The section highlighted in green is the MAC address of the ESXi host. To verify this, run the command esxcfg-nics -l

Resolution

To ensure cluster stability and prevent locking conflicts, the configuration of shared RDMs and shared disks must be identical across all nodes.

Refer the following document for detailed steps - Oracle RAC clustering setup

 

Powered by Wolken Software