vSphere Client intermittently becomes unresponsive and reports OutOfMemoryError

Products

• VMware vCenter Server 7.0

Issue/Introduction

vSphere Client becomes inaccessible intermittently and requires service restart to recover.

• Accessing vCenter service in ELM (linked mode) might return the error "Could not connect to one or more vCenter Server systems:https://<vcenter FQDN>:443/sdk"

• /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

[YYYY-MM-DDTHH:MM:SS] [ERROR] VLSI-I/O dispatcher-1         com.vmware.vise.util.concurrent.WorkerThread VLSI-I/O dispatcher-1 terminated with exception:  java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
        at org.apache.http.nio.util.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:52)
        at org.apache.http.nio.util.ExpandableBuffer.expandCapacity(ExpandableBuffer.java:108)
        at org.apache.http.nio.util.ExpandableBuffer.expand(ExpandableBuffer.java:121)
        at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:69)
        at org.apache.http.nio.protocol.BasicAsyncResponseConsumer.onContentReceived(BasicAsyncResponseConsumer.java:82)
        at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:147)
        at com.vmware.vim.vmomi.client.http.impl.InetAddressSniffingResponseConsumer.consumeContent(InetAddressSniffingResponseConsumer.java:54)
        at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
[YYYY-MM-DDTHH:MM:SS] [ERROR] linkedVcGroup-pool-2053      70004081 100125 200019 com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl      Failed to connect to VC https://<vcenter FQDN>:443/sdk (ID) java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
        at org.apache.http.util.Asserts.check(Asserts.java:46)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
        at com.vmware.vim.vmomi.client.http.impl.HttpAsyncExchange.invokeWithinScope(HttpAsyncExchange.java:83)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:31)

Environment

VMware vCenter Server 7.x

Cause

This issue is caused due to "host.PatchManager.ScanV2" task being spawned for each separate ESXi host when user scan for patches for a Cluster, Datacenter or vCenter. The task type "host.PatchManager.ScanV2" generates a XML as result (>2 Mb size)

 

Resolution

This issue is fixed in VMware vCenter Server 7.0 Update 3q. Please refer to the release notes for more information.

VMware vCenter Server 7.0 Update 3q Release Notes

In case you are unable to update the vCenter Server, please follow the below workaround.

Workaround:

As a workaround, we need to manually set the results of the tasks to null to avoid OutOfMemory error of vSphere UI. To do this, please follow the steps below:

• Download the attached script prune_PatchManager_Task_Results.sh and upload to the impacted vCenter Server to the /tmp folder. 

In case of connection failure, refer to Connecting to vCenter Server Virtual Appliance using WinSCP fails with the error: Received too large (1433299822 B) SFTP packet. Max supported packet size is 1024000 B

• Use SSH to connect to the VCSA and enter the Bash shell

• Change the file permission as executable

chmod +x /tmp/prune_PatchManager_Task_Results.sh

• Run the following command in the terminal

./tmp/prune_PatchManager_Task_Results.sh

• Restart the vsphere-ui service using the below command

service-control --restart vsphere-ui
 

To schedule the same script as Cron Job

• Save the script to a persistent location on the vCenter Server

• Run the below command to open the crontab configuration file:

crontab -e

• Append the crontab file with this following command:

<Minute> <hour> <day_of_the_month> <month> <day_of_week> /tmp/prune_PatchManager_Task_Results.sh

Below example will run the script after every 24 hours of interval:

02 17 * * * /tmp/prune_PatchManager_Task_Results.sh

 Which will run the script every day at 17:02 UTC

• Save the file and exit 

The changes take effect immediately and the script run at the next scheduled time.

Note: The above script performs a copy of the vpx_task table from vCenter server database (VCDB) and post that executes the cleanup of the specific column having the xml data. During this process, the copy of the table will be saved under "/exports/vcdb/" directory on the VCSA.

The above file can be cleaned up manually via logging in to the VCSA using ssh or can add the below action as Cron Job.

02 19 * * * find /exports/vcdb/vpx_task*.csv -type f -mtime +1 -exec rm {} \;

This will run the cleanup job every day at 19:02 UTC and remove files older than 1 day.

 

Additional Information

VMware vCenter Server 7.0 Update 3q Release Notes