The CARR script for NSX Manager certificates stops progressing in "Validating 'CCP' certificate"

Products

• VMware NSX

Issue/Introduction

Symptoms:

• User uses CARR script 1.20 to replace NSX manager certificates. The CARR script successfully applied fixes for CBM_CORFU and various transport node certificates. However, during the post-rotation validation phase, the process became unresponsive at the "Validating 'CCP' certificate" step:

All validations done
Using default validation config: ./validation_config.yaml 
Do you want script to fix all above problems now ? [Yes/No]: Yes 
Applying fix for 'CBM_CORFU' ... 
Applying fix for 'API' ... 
Applying fix for 'VIP' ... 
Applying fix for 'STALE-CERTIFICATES' ... 
Applying fix for 'APH_TN' ... 
Applying fix for 'APH_AR' ... 
All fixes have been made. 
Since transport node certificates were rotated, waiting 60s for messaging/clients API to update ... 
Validating one more time ... 
Validating 'VIP' certificate ... 
Validating 'STALE-CERTIFICATES' certificate ... 
Validating 'APH_AR' certificate ... 
Validating 'COMPUTE_MANAGER' certificate ... 
Validating 'API' certificate ... 
Validating 'SITE-TO-SITE' certificate ... 
Validating 'HOST' certificate ... 
Validating 'EDGE' certificate ... 
Validating 'CCP' certificate ...

Validating 'CCP' certificate .

 

• The NSX manager /var/log/cloudnet/nsx-ccp.log has error below:

YYYY-MM-DDTHH:MM:SS  WARN CCP-xxxxxx:boss-0 DefaultChannelPipeline 87995 An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open files
YYYY-MM-DDTHH:MM:SS  WARN CCP-xxxxxx:worker-0 ChannelInitializer 1481 Failed to initialize a channel. Closing: [id: xxxxxx, L:/<NSX manager IP>:1235 - R:/<ESXi IP>:58599]
java.lang.RuntimeException: Configuring Ssl threw
        at com.vmware.nsx.rpc.transport.netty.NettyServerChannelInitializer.configureAndAddSsl(NettyServerChannelInitializer.java:105) ~[libnsx_rpc.jar:?]
        at com.vmware.nsx.rpc.transport.netty.NettyServerChannelInitializer.initChannel(NettyServerChannelInitializer.java:83) ~[libnsx_rpc.jar:?]
        at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391) ~[netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.77.Final.jar:4.1.77.Final]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: java.nio.file.FileSystemException: /opt/vmware/ccp/etc/vnvp_privkey.pem: Too many open files

 

• The command nsxcli -c get controllers on an ESXi host returns no connectivity to the NSX Manager controllers, it indicates a failure in the communication channel between the host and the Central Control Plane (CCP).

Environment

VMware NSX 4

Cause

Open file limit is exceeded in NSX manager.

Resolution

Perform a rolling reboot of NSX managers.

Additional Information

To get CARR script please see KB below:

https://knowledge.broadcom.com/external/article/369034