After enabling Virtual Flash, VMware vCenter Server 5.5 consumes most of the CPU, memory, or disk I/O

search cancel

After enabling Virtual Flash, VMware vCenter Server 5.5 consumes most of the CPU, memory, or disk I/O

book

Article ID: 302185

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:

Virtual Flash (vFlash) is enabled in your vSphere Server environment.
Most of the CPU and memory is consumed by:
- vpxd.exe
- java.exe
Disk space is consumed rapidly because the VMware vCenter Inventory Service data folder increase in size (located at C:\Program Files\VMware\Infrastructure\Inventory Service).
Log into the vSphere Web Client fails with the below error:

Client is not authenticated to VMware Inventory Service - https://<VC>:10433

Note: Even if the login is successful, the whole inventory is not available in the vSphere Web Client.
Searching objects in vCenter Server via vSphere Client fails with this error:

Unable to connect to web services to execute query.Verify that the 'VMware VirtualCenter Management Webservices' service is running on https://VC:10433
The ds.log file, located at C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs, contains entries similar to:

ERROR com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl] Store eror processing atom feed:

com.vmware.vim.query.server.store.exception.StoreException: com.vmware.vim.query.server.store.exception.StoreException: org.w3c.dom.ls.LSException: XML docu

ment structures must start and end within the same entity.

at com.vmware.vim.query.server.store.impl.StoreImpl.updateVmomiPullAtomFeed(StoreImpl.java:2168)

at com.vmware.vim.query.server.store.impl.StoreImpl.update(StoreImpl.java:2008)

at com.vmware.vim.query.server.provider.AbstractAtomPullProviderBase.checkFeed(AbstractAtomPullProviderBase.java:144)

at com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl$FeedPump.run(ProviderManagerServiceImpl.java:763)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: com.vmware.vim.query.server.store.exception.StoreException: org.w3c.dom.ls.LSException: XML document structures must start and end within the sam

e entity.

at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryProcessor.call(AtomFeedProcessor.java:129)

at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryProcessor.call(AtomFeedProcessor.java:46)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

... 3 more

Caused by: org.w3c.dom.ls.LSException: XML document structures must start and end within the same entity.

at xDB_9_0_11.qP.a(xdb:340)

at xDB_9_0_11.qP.a(xdb:354)

at xDB_9_0_11.qP.a(xdb:138)

at xDB_9_0_11.qP.c(xdb:133)

at xDB_9_0_11.qP.parse(xdb:46)

at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryUtil.create(AtomFeedProcessor.java:549)

at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$PullEntryProcessor.processEntry(AtomFeedProcessor.java:308)

at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryProcessor.call(AtomFeedProcessor.java:101)

... 9 more

Caused by: xDB_9_0_11.Ex: XML document structures must start and end within the same entity.

::::1:900395:900394:XML document structures must start and end within the same entity.

at xDB_9_0_11.jD.a(xdb:333)

at xDB_9_0_11.jD.a(xdb:106)

at xDB_9_0_11.qP.a(xdb:352)

... 15 more

Caused by: ::::1:900395:900394:XML document structures must start and end within the same entity.

at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)

at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)

at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)

at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)

at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source)

at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)

at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)

at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)

at org.apache.xerces.impl.XMLEntityScanner.scanLiteral(Unknown Source)

at org.apache.xerces.impl.XMLScanner.scanAttributeValue(Unknown Source)

at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanAttribute(Unknown Source)

at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)

at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)

at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)

at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)

at xDB_9_0_11.jD.a(xdb:306)

... 17 more

The vpxd.log file, located at C:\ProgramData\VMware\VMware VirtualCenter\Logs, contains entries similar to:

[06236 error 'commonvpxVmomi' opID=3C35744C-001F520A-a8] [VpxVmomi] Exception while sending result: Not initialized: vim.host.FileSystemVolume volume
YYYY-02-04T07:54:16.157+01:00 [06236 verbose 'commonvpxVmomi' opID=3C35744C-001F520A-a8] backtrace[00] rip 000000018018cd7a
backtrace[01] rip 0000000180106c48
backtrace[02] rip 000000018010803e
backtrace[03] rip 00000001800907f8
backtrace[04] rip 0000000000524103
backtrace[05] rip 000000000059c307
backtrace[06] rip 00000000005a572c
backtrace[07] rip 00000000005a5be5
backtrace[08] rip 00000000005a654d
backtrace[09] rip 00000000005a68e7
backtrace[10] rip 00000000005a6a04
backtrace[11] rip 0000000000582407
backtrace[12] rip 00000000005833d6
backtrace[13] rip 000000013fd446d9
backtrace[14] rip 000000000055a527
backtrace[15] rip 000000000055cf80
backtrace[16] rip 00000000005663c9
backtrace[17] rip 0000000180154112
backtrace[18] rip 0000000180159031
backtrace[19] rip 000000018015a8b7
backtrace[20] rip 000000018015add8
backtrace[21] rip 000000018015398a
backtrace[22] rip 0000000180156269
backtrace[23] rip 0000000180155853
backtrace[24] rip 0000000180156793
backtrace[25] rip 0000000180156ad3
backtrace[26] rip 000000018019ef4a
backtrace[27] rip 0000000074052fdf
backtrace[28] rip 0000000074053080
backtrace[29] rip 0000000076d0652d
backtrace[30] rip 0000000076e3c521

Resolution

This is a known issue affecting VMware vCenter Server 5.5.

This issue is resolved in vCenter Server 5.5 Update 2, available at VMware Downloads. For more information, see the Resolved Issues section in the vCenter Server 5.5 Update 2 Release Notes.

Note: You may need to apply this workaround steps if there are affected hosts prior to applying vCenter Server 5.5 Update 2.

To verify your environment has affected hosts, query the database:

Connect the SQL Server where the vCenter Server's database resides using SQL Management Studio.
Query the vCenter Server database, with the command:

In vCenter Server:

select distinct fm.ID, h.DNS_NAME

from vpx_file_system_mount_info fm,

VPX_HOST h

where not exists

(select 1 from VPX_FS_MOUNT_VOLUME_INFO vi

where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID) and fm.ID = h.ID

In the VMware vCenter Server appliance with embedded database:

Log in to the VCSA as root.
Run this command:

cd /opt/vmware/vpostgres/9.0/bin

Run this command to connect to the VCDB database:

./psql -d VCDB -U vc

Run this command to query the vCenter Server database:

select distinct fm.ID, h.DNS_NAME from vpx_file_system_mount_info fm, VPX_HOST h where not exists (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID) and fm.ID = h.ID;

To quit, type:

\q

To work around this issue, execute the provided SQL commands on the vCenter Server database and modify the vmConfigInfo.xmlfile. Then restart vCenter Server.

To update the vCenter Server database configuration and modify vmConfigInfo.xml:

Use the query above to list the affected ESXi hosts from vCenter Server.
Stop the VirtualCenter Server service. For more information, see Stopping, starting, or restarting vCenter services (1003895).
Go to:
- In vCenter Server: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\
- VMware vCenter Server appliance: /usr/lib/vmware-vpx/
Take a backup of vmConfigInfo.xml.

Warning: Do not skip this step.
Open vmConfigInfo.xml in a text editor.
Insert the following information after the section that begins with after the
and before the section.

In this example the new data is added in red:

...

...

Save changes in vmConfigInfo.xml and close the file.
Take a backup of the vCenter Server SQL database.
Execute this query to update the vCenter Server database configuration:

In vCenter Server:

update VPX_HOST set MASTER_GEN=0, MASTER_SPEC_GEN=0
where ID in
(
select distinct fm.ID
from vpx_file_system_mount_info fm
where not exists
(select 1 from VPX_FS_MOUNT_VOLUME_INFO vi
where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID)
)
In the VMware vCenter Server appliance with embedded database:

cd /opt/vmware/vpostgres/9.0/bin
Run this command to connect to the VCDB database:

./psql -d VCDB -U vc
Run this command to update the vCenter Server database configuration:

update VPX_HOST set MASTER_GEN=0, MASTER_SPEC_GEN=0 where ID in (select distinct fm.ID from vpx_file_system_mount_info fm where not exists (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID));

Execute this query to update the vCenter Server database configuration:

In vCenter Server (running on Windows), connect to the SQL Server (running the vCenter Server database) and run this command:

delete from vpx_file_system_mount_info
where VPX_FILE_SYSTEM_MOUNT_INFO_ID in
(
select fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID
from vpx_file_system_mount_info fm left outer join VPX_FS_MOUNT_VOLUME_INFO vi
on fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID
where vi.FS_ID is NULL
)
In the VMware vCenter Server appliance with embedded database:
1. cd /opt/vmware/vpostgres/9.0/bin
2. Run this command to connect to the VCDB database:
  
  ./psql -d VCDB -U vc
3. Run this command to update the vCenter Server Appliance database configuration:
  
  delete from vpx_file_system_mount_info where VPX_FILE_SYSTEM_MOUNT_INFO_ID in (select fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID from vpx_file_system_mount_info fm left outer join VPX_FS_MOUNT_VOLUME_INFO vi on fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID where vi.FS_ID is NULL);

Restart vCenter Server to reflect changes.
After the vCenter Server is running, vCenter Server forces a fill sync with all affected hosts, updating the correct value.

Notes:

If the issue still persists, the query returns the ESXi host's Managed Object ID (MOID) as well as the DNS name. To resolve the issue, remove and re-add that host to the vCenter Server.

Caution: Do not remove the hosts from inventory if the hosts are used for vCloud Director or VMware Horizon View, removing the hosts causes virtual machines to become invalid. VMware recommends you to contact support for further troubleshooting.
If this does not resolve your issue, open a VMware Support Request. For more information, see Filing a Support Request in Customer Connect (2006985).

Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box How to stop, start, or restart vCenter Server services
How to file a Support Request in Customer Connect
Virtual Flash を有効化にした後、VMware vCenter Server 5.5 でほとんどの CPU、メモリまたはディスク I/O が消費される
启用虚拟闪存后，VMware vCenter Server 5.5 消耗大部分 CPU、内存或磁盘 I/O

Feedback

thumb_up Yes

thumb_down No