Intermittently vCenter server UI(vSphere client) stops working, showing an error as, “Could not connect to one or more vCenter Server systems”.
search cancel

Intermittently vCenter server UI(vSphere client) stops working, showing an error as, “Could not connect to one or more vCenter Server systems”.

book

Article ID: 419668

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Below log entries are observed in vpxd.log,

  • /var/log/vmware/vpxd/vpxd.log

yyyy-mm-ddThh:mm:ss.253Z info vpxd[2550999] [Originator@6876 sub=VmProv opID=lro-1-7f27cbea-01-01-01-01] Local-VC Datastore Migrate of poweredOn VM 'vm_name' (vm-20430, ds:///vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/vm_name/vm_name.vmx) on host-1050 (esxi_ip_address) in pool resgroup-1013 with ds ds:///vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/ to host-1050 (esxi_ip_address) in pool resgroup-1013 with ds ds:///vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/ with migId 1139555161061928491 as Operation: Local-VC_NonDRS_No-opvMotion
yyyy-mm-ddThh:mm:ss.260Z info vpxd[2550999] [Originator@6876 sub=VmCheck opID=lro-1-7f27cbea-01-01-01-01] CompatCheck results: (vim.vm.check.Result) [
..
yyyy-mm-ddThh:mm:ss.637Z error vpxd[2550999] [Originator@6876 sub=VmProv opID=lro-1-7f27cbea-01-01-01-01] Failed to track task vim.Task:task-583936 on host vim.HostSystem:host-1050: Fault cause: vim.fault.GenericVmConfigFault
-->
--> backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 8.0.3, build: build-24853646, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x00531C43]
--> backtrace[01] libvmacore.so[0x00421808]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)
--> backtrace[02] libvmacore.so[0x00433F29]: Vmacore::System::SystemFactory::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvmacore.so[0x004A9996]: Vmacore::Throwable::Throwable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)
--> backtrace[04] libvim-types.so[0x013A4553]
--> backtrace[05] libvim-types.so[0x013A637E]
--> backtrace[06] libvim-types.so[0x013A642C]
--> backtrace[07] libvim-types.so[0x013A6926]: Vim::Fault::GenericVmConfigFault::ThrowInternal()
--> backtrace[08] vpxd[0x012338DC]
--> backtrace[09] vpxd[0x02813FD5]
--> backtrace[10] vpxd[0x023C44F3]
--> backtrace[11] vpxd[0x023C48C2]
--> backtrace[12] vpxd[0x0236472D]
--> backtrace[13] vpxd[0x02403999]
--> backtrace[14] vpxd[0x024057DE]
--> backtrace[15] vpxd[0x02407A71]
--> backtrace[16] vpxd[0x0281795C]
--> backtrace[17] vpxd[0x028189A7]
--> backtrace[18] vpxd[0x02809C9B]
--> backtrace[19] vpxd[0x0280FA80]
--> backtrace[20] vpxd[0x02352841]
--> backtrace[21] vpxd[0x02746AB1]
--> backtrace[22] vpxd[0x027577C9]
--> backtrace[23] vpxd[0x0281795C]
--> backtrace[24] vpxd[0x028189A7]
--> backtrace[25] vpxd[0x02809C9B]
--> backtrace[26] vpxd[0x0281738A]
--> backtrace[27] libvmacore.so[0x0037EC04]
--> backtrace[28] libvmacore.so[0x00384517]
--> backtrace[29] libvmacore.so[0x00510FC5]
--> backtrace[30] libpthread.so.0[0x00008EB0]
--> backtrace[31] libc.so.6[0x000FFADF]
--> [backtrace end]

 

  • /var/run/log/hostd.log

yyyy-mm-ddThh:mm:ss48.566Z In(166) Hostd[2099219]: [Originator@6876 sub=Libs opID=lro-1-7f27cbea-01-01-01-01-b2-20ff sid=525efb5b user=vpxuser:<no user>] FILE: File_UnlinkRetry: /vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/vm_name/vm_name_1-flat.vmdk after 300 ms
yyyy-mm-ddThh:mm:ss52.869Z In(166) Hostd[2099219]: [Originator@6876 sub=Libs opID=lro-1-7f27cbea-01-01-01-01-b2-20ff sid=525efb5b user=vpxuser:<no user>] OBJLIB-FILEBE :FileBEUnlink : Failed to unlink the file '/vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/vm_name/vm_name_1-flat.vmdk' : 16
yyyy-mm-ddThh:mm:ss52.870Z In(166) Hostd[2099219]: [Originator@6876 sub=DiskLib opID=lro-1-7f27cbea-01-01-01-01-b2-20ff sid=525efb5b user=vpxuser:<no user>] DISKLIB-LIB   : Cannot remove extent '/vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/vm_name/vm_name_1-flat.vmdk': Device or resource busy
yyyy-mm-ddThh:mm:ss52.870Z Er(163) Hostd[2099219]: [Originator@6876 sub=DiskLib opID=lro-1-7f27cbea-01-01-01-01-b2-20ff sid=525efb5b user=vpxuser:<no user>] DISKLIB-LIB   : DiskLibUnlinkInt: Disk delete failed to complete { result:1048585, errMsg:'Device or resource busy', fileName:'/vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/vm_name/vm_name_1.vmdk'}
yyyy-mm-ddThh:mm:ss52.870Z Wa(164) Hostd[2099219]: [Originator@6876 sub=Vcsvc.OCM opID=lro-1-7f27cbea-01-01-01-01-b2-20ff sid=525efb5b user=vpxuser:<no user>] OCMCleanupHostlogItem: Unlink failed for /vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/vm_name/vm_name_1.vmdk: Device or resource busy
/vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/vm_name/vmware.log 
yyyy-mm-ddThh:mm:ss18.430Z In(05) vmx - SVMotion: Enter Phase 1
yyyy-mm-ddThh:mm:ss18.431Z Wa(03) worker-48081383 - Mirror: (digest)scsi1:0: SVMotionHandleExistingDisk: Failed: disk '/vmfs/volumes/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxx/vm_name/vm_name_1.vmdk' has a non-zero size
yyyy-mm-ddThh:mm:ss18.431Z Wa(03) worker-48081383 - Mirror: (digest)scsi1:0: SVMotionDiskSetup: failed to remove name reservation for destination disk.
yyyy-mm-ddThh:mm:ss18.431Z Wa(03) worker-48081383 - SVMotionPrepareForCopyThread: Failed to start SvMotion. Error during disk setup.
yyyy-mm-ddThh:mm:ss18.431Z In(05) worker-48081383 - SVMotion: FailureCleanup thread completes.

 

  • /var/run/log/vmkernel.log


yyyy-mm-ddThh:mm:ss27.039Z In(182) vmkernel: cpu11:2099219 opID=67836ea0)DLX: 2679: vol 'Datastore_name', lock at 173416448: Lock type: 10C00001. owner(s) MAC: yy:yy:yy:yy:yy:yy
yyyy-mm-ddThh:mm:ss27.039Z In(182) vmkernel: cpu11:2099219 opID=67836ea0)[type 10c00001 offset 173416448 v 800, hb offset 3571712
yyyy-mm-ddThh:mm:ss27Z In(182) vmkernel: gen 12093, mode 1, owner yyyyyyyy-yyyyyyyy-yyyy-yyyyyyyyyyy mtime 32381502
yyyy-mm-ddThh:mm:ss27Z In(182) vmkernel: num 0 gblnum 0 gblgen 0 gblbrk 0] alloc owner 0
yyyy-mm-ddThh:mm:ss27.039Z In(182) vmkernel: cpu11:2099219 opID=67836ea0)Fil3: 5174: Lock failed on file: vm_name_1-flat.vmdk on vol 'Datastore_name' with FD: <FD c10 r25>
yyyy-mm-ddThh:mm:ss27.342Z In(182) vmkernel: cpu11:2099219 opID=67836ea0)DLX: 4607: vol 'Datastore_name', lock at 173416448: [Req mode 1] Checking liveness:
yyyy-mm-ddThh:mm:ss27.342Z In(182) vmkernel: cpu11:2099219 opID=67836ea0)[type 10c00001 offset 173416448 v 800, hb offset 3571712
yyyy-mm-ddThh:mm:ss27Z In(182) vmkernel: gen 12093, mode 1, owner yyyyyyyy-yyyyyyyy-yyyy-yyyyyyyyyyy mtime 32381502

 

Environment

vCenter 8.x

Cause

 

This issue occurs as the disk file vm_name_1-flat.vmdk is locked or is being used by another process or task on the ESXi with MAC address pointed in vmkernel log file.

Resolution

Perform the steps to identify and clear the lock of the VM as mentioned in below document.

Device or resource busy/ file is locked error when making changes to a file on an ESXi datastore

Perform reboot of the host whose MAC is listed in vmkernel.log (ESXi which is holding the lock on VM file).

Powered by Wolken Software