After a vCenter Server reboot, VMware VirtualCenter Server service starts but vSphere Client sessions fail with the error: LOADING....
search cancel

After a vCenter Server reboot, VMware VirtualCenter Server service starts but vSphere Client sessions fail with the error: LOADING....

book

Article ID: 330139

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • After a vCenter Server reboot, vSphere Client connections fail with the error:

    LOADING...

  • After an extended period of time, you can connect to vCenter Server using the vSphere Client without an error.
  • You are using an Oracle database for vCenter Server.
  • In the vpxd.log file, you see entries similar to:

    Note: By default, the vpxd.log file is located at C:\ProgramData\VMware\VMware VirtualCenter\logs.

    YYYY-MM-DDT<time> [05672 warning 'MoLock' opID=HB-host-21612@7748-4a9176b3] ***WARNING*** Lock host-21612 mode EXCLUSIVE held for 1061 ms
    YYYY-MM-DDT<time> [05712 warning 'Default' opID=3f82a3d4-SWI-52e5d7ee] [VdbStatement] SQL execution took too long: INSERT INTO VPX_EVENT_ARG (EVENT_ID, ARG_ID, ARG_TYPE, ARG_DATA, OBJ_TYPE, OBJ_NAME, VM_ID, HOST_ID, COMPUTERESOURCE_ID, DATASTORE_ID, NETWORK_ID, NETWORK_TYPE, DVS_ID, DATACENTER_ID, RESOURCEPOOL_ID, FOLDER_ID, ALARM_ID, SCHEDULEDTASK_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    YYYY-MM-DDT<time> [06212 warning 'Default' opID=FdmMonitor-domain-c3201-5cd6f503] [VdbStatement] SQL execution took too long: SELECT DISTINCT B.UUID, CANONICAL_NAME FROM VPXV_DS_LUN_INFO A, VPX_SCSI_LUN B WHERE URL = ? AND A.UUID = B.UUID
    YYYY-MM-DDT<time> [06212 warning 'Default' opID=FdmMonitor-domain-c3201-5cd6f503] [VdbStatement] Execution elapsed time: 151211 ms
    YYYY-MM-DDT<time> [06240 warning 'Default' opID=HB-host-15652@64465-7f6e5e] [VdbStatement] SQL execution took too long: SELECT t1.NAME, MAX(t1.FILE_UNIQUESIZE), MAX(t1.FILE_SIZE) FROM VPX_VM_FLE_FILE_INFO t1, VPX_VM_FLE_FILE_INFO tmp WHERE tmp.VM_ID = ? AND (tmp.TYPE = 'diskDescriptor' OR tmp.TYPE = 'diskExtent') AND tmp.NAME = t1.NAME GROUP BY t1.NAME HAVING COUNT(t1.VM_ID) > 1
    YYYY-MM-DDT<time> [05368 warning 'Default' opID=HB-host-8401@28507-2872de8f] [VdbStatement] SQL execution took too long: SELECT t1.NAME, MAX(t1.FILE_UNIQUESIZE), MAX(t1.FILE_SIZE) FROM VPX_VM_FLE_FILE_INFO t1, VPX_VM_FLE_FILE_INFO tmp WHERE tmp.VM_ID = ? AND (tmp.TYPE = 'diskDescriptor' OR tmp.TYPE = 'diskExtent') AND tmp.NAME = t1.NAME GROUP BY t1.NAME HAVING COUNT(t1.VM_ID) > 1
    YYYY-MM-DDT<time> [05712 warning 'Default' opID=3f82a3d4-SWI-52e5d7ee] [VdbStatement] SQL execution took too long: INSERT INTO VPX_EVENT_ARG (EVENT_ID, ARG_ID, ARG_TYPE, ARG_DATA, OBJ_TYPE, OBJ_NAME, VM_ID, HOST_ID, COMPUTERESOURCE_ID, DATASTORE_ID, NETWORK_ID, NETWORK_TYPE, DVS_ID, DATACENTER_ID, RESOURCEPOOL_ID, FOLDER_ID, ALARM_ID, SCHEDULEDTASK_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    YYYY-MM-DDT<time> [06300 warning 'Default' opID=FdmMonitor-domain-c2008-68b692d8] [VdbStatement] SQL execution took too long: SELECT DISTINCT B.UUID, CANONICAL_NAME FROM VPXV_DS_LUN_INFO A, VPX_SCSI_LUN B WHERE URL = ? AND A.UUID = B.UUID
    YYYY-MM-DDT<time> [06228 warning 'Default' opID=FdmMonitor-domain-c4402-62740e4c] [VdbStatement] SQL execution took too long: SELECT DISTINCT B.UUID, CANONICAL_NAME FROM VPXV_DS_LUN_INFO A, VPX_SCSI_LUN B WHERE URL = ? AND A.UUID = B.UUID
    YYYY-MM-DDT<time> [06652 warning 'Default' opID=HB-host-15675@88643-29c1fc83] [VdbStatement] SQL execution took too long: SELECT DISTINCT B.UUID, CANONICAL_NAME FROM VPXV_DS_LUN_INFO A, VPX_SCSI_LUN B WHERE URL = ? AND A.UUID = B.UUID
    YYYY-MM-DDT<time> [03756 info 'vpxdvpxdInvtHostCnx' opID=HB-host-15212@1891-89b9ea7] [VpxdHostSync] Retrieved host update to 1891 for host-15212
    YYYY-MM-DDT<time> [03756 info 'vpxdvpxdInvtHostDb' opID=HB-host-15212@1891-89b9ea7] [VpxdInvtHost::SaveFieldsToDb] IPMI info of host gbahevp52.ics.express.tnt is not set
    YYYY-MM-DDT<time> [05328 warning 'Default' opID=cec53e7-SWI-11a2baed] [VdbStatement] SQL execution took too long: INSERT INTO VPX_EVENT_ARG (EVENT_ID, ARG_ID, ARG_TYPE, ARG_DATA, OBJ_TYPE, OBJ_NAME, VM_ID, HOST_ID, COMPUTERESOURCE_ID, DATASTORE_ID, NETWORK_ID, NETWORK_TYPE, DVS_ID, DATACENTER_ID, RESOURCEPOOL_ID, FOLDER_ID, ALARM_ID, SCHEDULEDTASK_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    2014-09-05T15:18:57.373+01:00 [05328 warning 'Default' opID=cec53e7-SWI-11a2baed] [VdbStatement] Execution elapsed time: 4072 ms
    2014-09-05T15:18:57.373+01:00 [05328 warning 'Default' opID=cec53e7-SWI-11a2baed] [VdbStatement] Bind parameters:
    2014-09-05T15:18:57.373+01:00 [05328 warning 'Default' opID=cec53e7-SWI-11a2baed] [VdbStatement] datatype: 1, size: 4, arraySize: 149</time></time></time></time></time></time></time></time></time></time></time></time></time>


Cause

This issue occurs when the PGA memory (Program Global Area) on the Oracle database (where the vCenter Server database is located) is too low.

Resolution

To resolve this issue, increase the PGA memory size on the Oracle database in which your vCenter Server database is located.
For more information, see the Tuning the Program Global Area section of the Oracle Database Administration Guide.

Note: The preceding link was correct as of January 1, 2015. If you find the link is broken, provide feedback and a VMware employee will update the link.

Note: VMware recommends engaging your Oracle database Administrator to carry out this procedure.


Additional Information

PGA is a memory region that contains data and control information for a server process. For more information, see the Memory Architecture section of the Oracle Database Administration Guide.

Note: The preceding link was correct as of January 1, 2015. If you find the link is broken, provide feedback and a VMware employee will update the link.

From the performance reports on an Oracle database, if you see that the biggest wait times are for the read and writes to the temporary data file, then this indicates that the PGA size is set too low and must be resized to a larger value.
Note: This is an issue during a reboot of vCenter Server, but not during normal running process.
For example, this is an Oracle performance report with high read and write wait times:

Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
Event Waits Time(s) Avg wait(ms) % DB Time
----------------------------------------- ------------ ----------- ------------------ -----------
direct path read temp 1,876,465 14,476 8 62.4
direct path write temp 219,404 5,844 27 25.2
CPU time 1,111 4.8
control file sequential read 5,356 364 68 1.6
control file parallel write 276 355 1286 1.5
------------------------------------------------------------------------------------------------------

For the preceding report, PGA must be resized.


Powered by Wolken Software