VMware vCenter Server 5.1/5.5 fails after hot-adding a shared VMDK or RDM
search cancel

VMware vCenter Server 5.1/5.5 fails after hot-adding a shared VMDK or RDM

book

Article ID: 338802

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • After hot-adding a virtual disk that is already being used by another virtual machine with bus sharing enabled, you see this error:

    The VM failed to resume on the destination during early power on.
    Module DiskEarly power on failed.
    Cannot open the disk '/vmfs/volumes/datastoreUUID/vmdkname.vmdk' or one of the snapshot disks it depends on.
    Failed to lock the file
    Hot-add of scsi0, scsi0:0 failed.
    The hot-plug operation failed.

  • Attempting to edit the virtual machine setting in the vSphere Client, you see this error:

    A Flat backing option was not found.
    An unknown type of controller is found in the virtual machine.

  • In the vCenter Server C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd.log file, you see entries similar to:

    <YYYY-MM-DD>T<time> [04968 error 'VmProv' opID=3AF4331D-0000010D-df opName=create vmName=Bus Sharing (Node B)] Unexpected exception (vim.fault.GenericVmConfigFault) while reconfiguring VM. Aborting.</time>
    <YYYY-MM-DD>T<time> [04968 error 'VmProv' opID=3AF4331D-0000010D-df opName=create vmName=Bus Sharing (Node B)] Backtrace: backtrace[00] rip 000000018018b7fa </time>
    --> backtrace[01] rip 0000000180104c78
    --> backtrace[02] rip 0000000180105f6e
    --> backtrace[03] rip 000000018008ff08
    --> backtrace[04] rip 0000000001412bd1
    --> backtrace[05] rip 000007f788198c64
    --> backtrace[06] rip 000007f78823e6a0
    --> backtrace[07] rip 000007f78821ae7c
    --> backtrace[08] rip 000007f78820bf0e
    --> backtrace[09] rip 000007f787f3a9d2
    --> backtrace[10] rip 000007f787f3b94a
    --> backtrace[11] rip 000007f787f41cce
    --> backtrace[12] rip 000007f787f45282
    --> backtrace[13] rip 00000000011a5637
    --> backtrace[14] rip 0000000000d6980c
    --> backtrace[15] rip 000007f7881a12c6
    --> backtrace[16] rip 000007f7881791ff
    --> backtrace[17] rip 000007f78817ce14
    --> backtrace[18] rip 000007f78818baf4
    --> backtrace[19] rip 000007f788194575
    --> backtrace[20] rip 0000000180152544
    --> backtrace[21] rip 00000001801525d1
    --> backtrace[22] rip 0000000180156eec
    --> backtrace[23] rip 00000001801587ba
    --> backtrace[24] rip 0000000180158cfc
    --> backtrace[25] rip 0000000180158e98
    --> backtrace[26] rip 0000000180153ab3
    --> backtrace[27] rip 0000000180154b03
    --> backtrace[28] rip 0000000180154e43
    --> backtrace[29] rip 000000018019d62a
    --> backtrace[30] rip 000000006b0c2fdf
    --> backtrace[31] rip 000000006b0c3080
    --> backtrace[32] rip 000007fe6e0c1842
    --> backtrace[33] rip 000007fe6e96f9d1
    -->
    ~
    <YYYY-MM-DD>T<time> [04968 warning 'Default' opID=3AF4331D-0000010D-df] [JrnlFilePersistenceProvider] Got EOF while reading file</time>
    <YYYY-MM-DD>T<time> [04968 info 'vmprovvpxdVmprovXaction' opID=3AF4331D-0000010D-df] [VpxdVmprovXaction] Starting journal rollback</time>
    <YYYY-MM-DD>T<time> [04968 info 'vmprovvpxdVmprovXaction' opID=3AF4331D-0000010D-df] [VpxdVmprovXaction] Vm provision transaction rollback done</time>
    <YYYY-MM-DD>T<time> [04968 error 'vmprovvpxdVmprov' opID=3AF4331D-0000010D-df] [VpxdVmprov::Reconfigure] Unexpected exception during reconfig</time>
    <YYYY-MM-DD>T<time>[04968 info 'commonvpxLro' opID=3AF4331D-0000010D-df] [VpxLRO] -- FINISH task-1340 -- vm-142 -- vim.VirtualMachine.reconfigure --</time>
    ~
    <YYYY-MM-DD>T<time> [04972 error 'VmValidate' opID=task-internal-1-bc5816b-5c-de testSet=host-29,vm-142] Unexpected NULL controllerType encountered</time>
    <YYYY-MM-DD>T<time> [04972 error 'VmValidate' opID=task-internal-1-bc5816b-5c-de testSet=host-32,vm-142] Unexpected NULL controllerType encountered</time>
    ~
    <YYYY-MM-DD>T<time> [21048 panic 'Default' opID=task-internal-1-1bcf7561-3f-ff testSet=host-32,vm-142] </time>
    -- >
    -- > Panic: Assert Failed: "controllerType != 0" @ d:/build/ob/bora-1750787/bora/vim/lib/vmvalidate/configOptionValidate.cpp:558
    ~
    <YYYY-MM-DD>T<time>[04200 warning 'vpxdvpxdService'] Service control manager requested shutdown</time>
    <YYYY-MM-DD>T<time>[04200 info 'ThreadPool'] Thread enlisted</time>
    <YYYY-MM-DD>T<time>[04200 info 'Default'] Initiating VMware VirtualCenter shutdown</time>
    <YYYY-MM-DD>T<time> [02320 info 'Default'] Shutting down VMware VirtualCenter...

    </time>
  • The vCenter Server vpxd.log file may also contain similar errors after repeated attempts to restart:
<YYYY-MM-DD>T<time> [15084 error 'Default'] A database error occurred: ODBC error: (08001) - [Microsoft][SQL Server Native Client 10.0]Unable to complete login process due to delay in opening server connection</time>
<YYYY-MM-DD>T<time> [15084 warning 'vpxdvpxdMain'] Database not initialized. Nothing to unlock</time>
<YYYY-MM-DD>T<time>0 [15084 info 'Default'] Forcing shutdown of VMware VirtualCenter now
</time>
  • The /var/log/hostd.log file on the ESXi host in which the virtual machine resides contains errors similar to:

    <YYYY-MM-DD>T<time>Z [3A2C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/datastoreUUID/vmdkname.vmdk'] Commit Vigor hot-plug batch 1298299155 translated error to vim.fault.GenericVmConfigFault</time>
    <YYYY-MM-DD>T<time>Z [3A2C1B70 info 'Vmsvc.vm:/vmfs/volumes/datastoreUUID/vmdkname.vmdk'] Commit Vigor hot-plug batch 1298299155 failed: vim.fault.GenericVmConfigFault</time>
    <YYYY-MM-DD>T<time>Z [3A2C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/datastoreUUID/vmdkname.vmdk'] Commit Vigor hot-plug batch 1298299155 message: The hot-plug operation failed.</time>
    --> Hot-add of scsi0, scsi0:0 failed.
    --> Failed to lock the file
    --> Cannot open the disk '/vmfs/volumes/datastoreUUID/vmdkname.vmdk' or one of the snapshot disks it depends on.
    --> Module DiskEarly power on failed.
    --> The VM failed to resume on the destination during early power on.
    -->
    <YYYY-MM-DD>T<time>Z [3A2C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/datastoreUUID/vmdkname.vmdk'] Signalling Commit Vigor hot-plug batch 1298299155</time>
    <YYYY-MM-DD>T<time>Z [39E40B70 verbose 'Vmsvc.vm:/vmfs/volumes/datastoreUUID/vmdkname.vmdk' opID=3AF4331D-0000010D-df-37 user=vpxuser] Completed Commit Vigor hot-plug batch 1298299155</time>
    <YYYY-MM-DD>T<time>Z [39E40B70 verbose 'Vmsvc.vm:/vmfs/volumes/datastoreUUID/vmdkname.vmdk' opID=3AF4331D-0000010D-df-37 user=vpxuser] Commit Vigor hot-plug batch 1298299155 re-throwing vim.fault.GenericVmConfigFault</time>
    <YYYY-MM-DD>T<time>Z [39E40B70 info 'Vmsvc.vm:/vmfs/volumes/datastoreUUID/vmdkname.vmdk' opID=3AF4331D-0000010D-df-37 user=vpxuser] Reconfigure failed: vim.fault.GenericVmConfigFault

    </time>
  • In the virtual machine's vmware.log file, you see entries similar to:

    <YYYY-MM-DD>T<time></time>Z| vmx| I120: DISK: Cannot open disk "/vmfs/volumes/datastoreUUID/vmdkname.vmdk": Failed to lock the file (16392).
    <YYYY-MM-DD>T<time>Z| vmx| I120: DISK: Opening disks took 25606 ms.</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: Module DiskEarly power on failed.</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: VMX_PowerOn: ModuleTable_PowerOn = 0</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: MigrateSetStateFinished: type=2 new state=11</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: MigrateSetState: Transitioning from state 10 to 11.</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: [msg.migrate.resume.fail] The VM failed to resume on the destination during early power on.</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: Msg_Post: Error</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: [msg.fileio.lock] Failed to lock the file</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/datastoreUUID/vmdkname.vmdk' or one of the snapshot disks it depends on.</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: [msg.moduletable.powerOnFailed] Module DiskEarly power on failed.</time>
    <YYYY-MM-DD>T<time>Z| vmx| I120: [msg.migrate.resume.fail] The VM failed to resume on the destination during early power on.

    </time>
    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware vCenter Server 5.5.x
VMware vCenter Server 5.1.x

Cause

This issues occurs when you attempt to hot-add a VMDK or RDM to a powered on virtual machine, a new SCSI controller is created or the existing SCSI controller is reconfigured for Bus Sharing. For shared disks, Bus Sharing must be configured on each virtual machine and can only be enabled or disabled while a virtual machine is powered off.

Resolution

This is a known issue affecting VMware vCenter Server 5.1 and 5.5.

This issue is resolved in:

To work around this issue:
  1. Stop the VMware VirtualCenter Server service. For more information, see Stopping, starting, or restarting VMware vCenter Server services (1003895).

  2. To determine which virtual machines are impacted, run this command on each host in the environment:

    find /vmfs/volumes/ -iname "*.vmx" -exec grep -Hie "scsi[0-9].present = \"FALSE\"" {} \;

    Note    : Run this command on each ESXi host. Virtual machine configuration files in-use by other hosts are not readable by others when running this command.

  3. Connect directly to the host in which the virtual machine resides, power down and remove the virtual machine from inventory.
  4. If you are unable to connect to the host using the vSphere Client to power down the virtual machine, use these steps to identify and power down the affected virtual machine:

    1. To list the virtual machines currently registered on the host, run this command:

      vim-cmd vmsvc/getallvms

      You see output similar to:
      vim-cmd vmsvc/getallvms

      Vmid Name File Guest OS Version Annotation
      112 VM-1 [Datastore] VM-3/VM-3.vmx rhel16_64Guest vmx-04

    2. Using the output from the previous step, locate the affected virtual machine and make note of the Vmid.
    3. If the virtual machine is currently powered on, run this command:

      VMware Tools is installed on the virtual machine:

      vim-cmd vmsvc/power.shutdown vmid

      If VMware Tools is not installed, run this command:

      vim-cmd vmsvc/power.off vmid
    4. Run this command to unregister the virtual machine from the host's inventory:

      vim-cmd vmsvc/unregister       vmid


  5. Start the VMware VirtualCenter Server service. For more information, see Stopping, starting, or restarting VMware vCenter Server services (1003895).
  6. The affected virtual machine appears as orphaned with vCenter Server. Remove this virtual machine from inventory.
  7. Create a new virtual machine using the previous settings of the recently deleted virtual machine. Since the virtual machine is powered off, the virtual machine with shared VMDKs or RDMs should be created successfully.


Additional Information

How to stop, start, or restart vCenter Server services
共有 VMDK または RDM をホットアドした後、VMware vCenter Server 5.1/5.5 が失敗する
在热添加共享 VMDK 或 RDM 后 VMware vCenter Server 5.1/5.5 失败

Powered by Wolken Software