vc-ws1a-broker Service Stops Causing vCenter Backup Failures giving "Too many open files" Error

Products

• VMware vCenter Server

Issue/Introduction

• The ws1a-broker service stops running unexpectedly due to which vCenter Backups fail.
• The /var/log/vmware/vc-ws1a-broker/federation-service.log will gives the below error:

  Caused by: beans.BeanInstantiationException: Failed to instantiate [com.vmware.vidm.federation.token.TokenPersistenceService]: Constructor threw exception; nested exception is java.lang.IllegalStateException: failed to create a child event loop
          at beans.BeanUtils.instantiateClass(BeanUtils.java:226)
          at beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
          at beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:302)
          ... 81 more
  Caused by: java.lang.IllegalStateException: failed to create a child event loop
          at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
          at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
      
  Caused by: io.netty.channel.ChannelException: failed to open a new selector
          at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:179)
          at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:146)
          at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:183)
          at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:38)
          at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
          ... 102 more
  Caused by: java.io.IOException: Too many open files
          at java.base/sun.nio.ch.EPoll.create(Native Method)
          at java.base/sun.nio.ch.EPollSelectorImpl.<init>(Unknown Source)
          at java.base/sun.nio.ch.EPollSelectorProvider.openSelector(Unknown Source)
          at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:177)
          ... 106 more
  

  
  


 

Environment

VMware vCenter Server 8.0.x

Cause

The "Too many open files" error occurs when a process exceeds its maximum allowed number of open file descriptors. By default, this limit is restricted to 1024 in the service configuration.

Resolution

To resolve this issue, you need to increase the resource limits (rlimits) for the vc-ws1a-broker container from 1024 to 2048.

1. SSH to vCenter and manually start the service

   service-control --start vc-ws1a-broker

2. Identify bundle location for the service using the command 'runc list'.

     runc list
     ID               PID         STATUS      BUNDLE     
     vc-ws1a-broker   12636       running     /storage/containers/vc-ws1a-broker/#############################################  

3. Open the config.json file from the path obtained in step 2

   vi /storage/containers/vc-ws1a-broker/#############################################/config.json

4. Change the values for 'rlimit' in the config file from 1024 to 2048 using VI editor.

    "noNewPrivileges": true,
             "rlimits": [
                 {       
                     "hard": 1024,
                     "soft": 1024,
                     "type": "RLIMIT_NOFILE"
                 }   

5. Type :wq! to save and close the editor
6. Restart the vc-ws1a-broker service

   service-control --restart vc-ws1a-broker

7. Login to the container to check the new limits. It should look like this:

     runc exec --tty vc-ws1a-broker bash
     idmservice [ / ]$ ulimit -Hn
     2048
     idmservice [ / ]$ ulimit -Sn
     2048

   Applying these changes resolves the "Too many open files" error, which occurs when the service exhausts its default allocation of 1024 file descriptors, allowing your backups to run successfully.