Service vc-ws1a-broker Fails to Start With Error: "Too many open files"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vc-ws1a-broker service is in stopped state and fails to start
• The /var/log/vmware/vc-ws1a-broker/federation-service.log will denote the following
  Caused by: java.io.IOException: Too many open files

 

Environment

VMware vCenter Server 8.0.x

Cause

The "Too many open files" error happens when a process needs more open files than the limit. By default, it is set to 1024 in the service configuration.

Resolution

Increase the file descriptors maximum for the vc-ws1a-broker container

1. SSH to vCenter and manually start the service
  # service-control --start vc-ws1a-broker
2. Identify bundle location for the service using the command 'runc list'
  # runc list
  ID               PID         STATUS      BUNDLE         CREATED         OWNER
  vc-ws1a-broker   12636       running     /storage/containers/vc-ws1a-broker/############################################# 2024-04-26T19:37:49.874571921Z   root
3. Open the config.json file from the path obtained in step 2
   #  vi /storage/containers/vc-ws1a-broker/#############################################/config.json
4. Change the values for 'rlimit' in the config file from 1024 to 2048
  "noNewPrivileges": true,
          "rlimits": [
              {       
                  "hard": 1024,
                  "soft": 1024,
                  "type": "RLIMIT_NOFILE"
              }   
5. Type :wq! to save and close the editor
6. Restart the vc-ws1a-broker service
  # service-control --restart vc-ws1a-broker
7. Login to the container to check the new limits. It should look like this:
  # runc exec --tty vc-ws1a-broker bash
  idmservice [ / ]$ ulimit -Hn
  2048
  idmservice [ / ]$ ulimit -Sn
  2048