The CARR script for NSX Manager certificates stops progressing in "Validating 'CCP' certificate"
Article ID: 425372
Updated On:
Products
Issue/Introduction
Symptoms:
- User uses CARR script 1.20 to replace NSX manager certificates. The CARR script successfully applied fixes for CBM_CORFU and various transport node certificates. However, during the post-rotation validation phase, the process became unresponsive at the "Validating 'CCP' certificate" step:
All validations doneUsing default validation config: ./validation_config.yaml Do you want script to fix all above problems now ? [Yes/No]: Yes Applying fix for 'CBM_CORFU' ... Applying fix for 'API' ... Applying fix for 'VIP' ... Applying fix for 'STALE-CERTIFICATES' ... Applying fix for 'APH_TN' ... Applying fix for 'APH_AR' ... All fixes have been made. Since transport node certificates were rotated, waiting 60s for messaging/clients API to update ... Validating one more time ... Validating 'VIP' certificate ... Validating 'STALE-CERTIFICATES' certificate ... Validating 'APH_AR' certificate ... Validating 'COMPUTE_MANAGER' certificate ... Validating 'API' certificate ... Validating 'SITE-TO-SITE' certificate ... Validating 'HOST' certificate ... Validating 'EDGE' certificate ... Validating 'CCP' certificate ...
Validating 'CCP' certificate .
- The NSX manager /var/log/cloudnet/nsx-ccp.log has error below:
YYYY-MM-DDTHH:MM:SS WARN CCP-xxxxxx:boss-0 DefaultChannelPipeline 87995 An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open filesYYYY-MM-DDTHH:MM:SS WARN CCP-xxxxxx:worker-0 ChannelInitializer 1481 Failed to initialize a channel. Closing: [id: xxxxxx, L:/<NSX manager IP>:1235 - R:/<ESXi IP>:58599]java.lang.RuntimeException: Configuring Ssl threw at com.vmware.nsx.rpc.transport.netty.NettyServerChannelInitializer.configureAndAddSsl(NettyServerChannelInitializer.java:105) ~[libnsx_rpc.jar:?] at com.vmware.nsx.rpc.transport.netty.NettyServerChannelInitializer.initChannel(NettyServerChannelInitializer.java:83) ~[libnsx_rpc.jar:?] at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) ~[netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.77.Final.jar:4.1.77.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.77.Final.jar:4.1.77.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391) ~[netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[netty-common-4.1.77.Final.jar:4.1.77.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.77.Final.jar:4.1.77.Final] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]Caused by: java.nio.file.FileSystemException: /opt/vmware/ccp/etc/vnvp_privkey.pem: Too many open files
- The command
nsxcli -c get controllerson an ESXi host returns no connectivity to the NSX Manager controllers, it indicates a failure in the communication channel between the host and the Central Control Plane (CCP).
Environment
VMware NSX 4
Cause
Open file limit is exceeded in NSX manager.
The SSL handshake is being blocked at the server side due to network packet loss. Consequently, the server is not closing the connection, resulting in stale connections.
Resolution
Restart the app proxy on the affected manager(s)
systemctl stop nsx-appl-proxy
systemctl start nsx-appl-proxy
Additional Information
To get CARR script please see KB below:
Feedback
