vSphere UI reports intermittent crash with OutOfMemoryError due to PatchManager.ScanV2 tasks getting queued up

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vSphere Client becomes inaccessible intermittently and requires service restart to recover

• Accessing vCenter service in ELM (linked mode) might return the error "Could not connect to one or more vCenter Server systems:https://<vcenter FQDN>:443/sdk"

• /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

[2023-09-18T13:08:40.010Z] [ERROR] VLSI-I/O dispatcher-1         com.vmware.vise.util.concurrent.WorkerThread VLSI-I/O dispatcher-1 terminated with exception:  java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
        at org.apache.http.nio.util.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:52)
        at org.apache.http.nio.util.ExpandableBuffer.expandCapacity(ExpandableBuffer.java:108)
        at org.apache.http.nio.util.ExpandableBuffer.expand(ExpandableBuffer.java:121)
        at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:69)
        at org.apache.http.nio.protocol.BasicAsyncResponseConsumer.onContentReceived(BasicAsyncResponseConsumer.java:82)
        at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:147)
        at com.vmware.vim.vmomi.client.http.impl.InetAddressSniffingResponseConsumer.consumeContent(InetAddressSniffingResponseConsumer.java:54)
        at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
[2023-09-18T14:02:19.789Z] [ERROR] linkedVcGroup-pool-2053      70004081 100125 200019 com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl      Failed to connect to VC https://<vcenter FQDN>:443/sdk (ID) java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
        at org.apache.http.util.Asserts.check(Asserts.java:46)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
        at com.vmware.vim.vmomi.client.http.impl.HttpAsyncExchange.invokeWithinScope(HttpAsyncExchange.java:83)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:31)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere 7.0.x

Cause

This issue is caused due to "host.PatchManager.ScanV2" task being spawned for each separate ESXi host    when user scan for patches for a Cluster, Datacenter or vCenter. The task type "host.PatchManager.ScanV2" generates a XML as result (>2 Mb size)

 

Resolution

VMware is aware of this issue and working to resolve this in a future release

Workaround:

As a workaround, we need to manually set the results of the tasks to null to avoid OutOfMemory error of vSphere UI. To do this, please follow the steps below:

• Download the attached script prune_PatchManager_Task_Results.sh and upload to the impacted vCenter Server to the /tmp folder. 

In case of connection failure, refer to Error when uploading files to vCenter Server Appliance using WinSCP

• Use SSH to connect to the VCSA and enter the Bash shell

• Change the file permission as executable

chmod +x /tmp/prune_PatchManager_Task_Results.sh

• Run the following command in the terminal

./tmp/prune_PatchManager_Task_Results.sh

• Restart the vsphere-ui service using the below command

service-control --restart vsphere-ui
 

To schedule the same script as Cron Job

• Save the script to a persistent location on the vCenter Server

• Run the below command to open the crontab configuration file :

crontab -e

• Append the crontab file with this following command :

<Minute> <hour> <day_of_the_month> <month> <day_of_week> /tmp/prune_PatchManager_Task_Results.sh

Below example will run the script after every 24 hours of interval :

02 17 * * * /tmp/prune_PatchManager_Task_Results.sh

 Which will run the script everyday at 17:02 UTC

• Save the file and exit 

The changes take effect immediately and the script run at the next scheduled time.

Note: The above script performs a copy of the vpx_task table from vCenter server database (VCDB) and post that executes the cleanup of the specific column having the xml data. During this process, the copy of the table will be saved under "/exports/vcdb/" directory on the VCSA.

The above file can be cleaned up manually via logging in to the VCSA using ssh or can add the below action as Cron Job.

02 19 * * * find /exports/vcdb/vpx_task*.csv -type f -mtime +1 -exec rm {} \;

which will run the cleanup everyday at 19:02 UTC and cleanup file older than 1 day