HTTP service stop working on NSX-T Manager nodes

Products

• VMware NSX Networking

Issue/Introduction

The purpose of this article is to highlight issue affecting NSX-T 2.4.x version where the HTTP service can crash randomly.

Symptoms:

• Web interface of one or more NSX-T Manager appliance are not reachable.
• In the proxy/catalina.log, you could see log messages similar to:

2019-09-02T23:25:09.245Z INFO org.apache.coyote.AbstractProtocol init Initializing ProtocolHandler ["https-jsse-nio-10.200.10.15-443"]
2019-09-02T23:25:09.257Z SEVERE org.apache.catalina.core.StandardService initInternal Failed to initialize connector [Connector[HTTP/1.1-443]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-443]]
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:552)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:875)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:632)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:655)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.tanukisoftware.wrapper.WrapperStartStopApp.run(WrapperStartStopApp.java:316)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:995)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        ... 18 more
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:219)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1087)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:265)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:993)
        ... 19 more

• In the proxy/proxy-tomcat-wrapper.log, you could see log messages similar to:

ERROR  | wrapper  | 2019/07/12 22:15:24 | JVM exited unexpectedly.
ERROR  | wrapper  | 2019/09/02 23:25:43 | Shutdown failed: Timed out waiting for signal from JVM.
STATUS | wrapper  | 2019/09/02 23:25:43 | Dumping JVM state.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center 2.x
VMware NSX-T Data Center

Resolution

This issue is resolved in NSX-T Data Center 2.5.

Workaround:
Two workarounds are possible:

• Restart the impacted node.
• Restart the HTTP service on the affected node: "start service http"