vCenter failing to update/patching due to vc-ws1a-broker service being overloaded

Products

• VMware vCenter Server

Issue/Introduction

• The vc-ws1a-broker service will not start
• It crashes an update or patching right at the end
  

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 9.0.x

Cause

/var/log/vmware/vc-ws1a-broker/federation-service.log

Caused by: java.io.IOException: Too many open files

This means that vc-ws1a-broker process has the max file descriptors for the process and can't open any more.

Resolution

Need to increase the size of the rlimit on the vc-ws1a-broker service.

You cannot do this before the the upgrade/patching as it will be over written with original configuration. You need to wait for the upgrade to either fail in the screenshot above or just before this when your see it hung at certain point in the upgrade/patching.

Steps to increase the fd limits for container vc-ws1a-broker

Step 1:

root@xxx-xxxx-xxx [ ~ ]# runc list
ID PID STATUS BUNDLE CREATED OWNER
vc-ws1a-broker 12636 running /storage/containers/vc-ws1a-broker/7137719fxxxxxxxxxxxxxxxxxxxxxxxxxxxb57e9 2024-04-26T19:37:49.874571921Z root

Sometime the above command will show nothing but you can still cd to /storage/containers/vc-ws1a-broker/ and you will see the folder

Step 2:

Open the file from the path above
vi /storage/containers/vc-ws1a-broker/7137719fxxxxxxxxxxxxxxxxxxxxxxxb57e9 /config.json

Step 3:

Change the values in this file from 1024 to 2048, and save the file

       "noNewPrivileges": true,
        "rlimits": [
            {
                "hard": 1024,
                "soft": 1024,
                "type": "RLIMIT_NOFILE"
           }

Step 4:

restart the vidb container - service-control --restart vc-ws1a-broker

Step 5:

Login to container to check the new limits, it should show like this.

root@xxx-xxxx-xxx [ ~ ]# runc exec --tty vc-ws1a-broker bash
idmservice [ / ]$ ulimit -Hn
2048
idmservice [ / ]$ ulimit -Sn
2048


You can not let the upgrade continue or click "Resume vCenter Update"

 

If still having issues please open a ticket with the Broadcom support team.