During an NSX-T Manager restore from backup, HTTPS service fails to start

Products

• VMware NSX

Issue/Introduction

During an NSX-T Manager restore from backup, if the configured VIDM server is unreachable, the restore process halts and HTTPS service does not start.

Symptoms:

• NSX-T Data Center environment with VIDM configured.
• A restore from FTP backup is being performed.
• HTTPS service is failing to start

>get cluster status
Cluster Id: b74a2dfd-####-####-####-68c6ced93ed5
Overall Status: DEGRADED

...........

Group Type: HTTPS
Group Status: UNAVAILABLE

Members:
    UUID                                       FQDN                                       IP               STATUS
    73060e42-####-####-####-2d78b38971be       Manager1                                192.168.2.1         DOWN

- Log signatures of this issue may be similar to these examples.

 /var/log/nvpapi/api_server.log
NSX-T 2.5 - 2021-01-05T01:22:11.121Z napi.root.node.aaa.providers.vidm.__self__ ERROR Error reaching given VMware Identity Manager address vidm_server.local.net | [Errno -2] Name or service not known | -2”
NSX-T 3.x - 2021-07-17T06:57:24.022Z napi.root.node.aaa.providers.vidm.__self__ ERROR Error connecting to VMware Identity Manager vidm_server.local.net | POST | /SAAS/auth/oauthtoken | [Errno 110] Connection timed out | 110


 /var/log/proton/nsxapi.log
2020-12-30T05:13:12.218Z WARN ccp-fullsync-task-thread1 AbstractFullSyncCommunicator - - [nsx@6876 comp="nsx-manager" level="WARN" subcomp="manager"] Full sync response was suppressed, bacause there is an ongoing Restore process. Application = com.vmware.nsx.messaging.StateSyncService2, clientId = 4c47843b-#####-#####-####-707599872e63.


 /var/log/proxy/catalina.log
2021-07-17T07:00:37.378Z SEVERE org.apache.catalina.core.StandardService initInternal Failed to initialize connector [Connector[com.vmware.nsx.management.security.ReloadProtocol-443]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1077)

Environment

VMware NSX-T Data Center

Resolution

This issue is resolved in NSX-T Data Center 3.1.2, available at VMware Downloads.

Workaround:
The workaround involves editing an xml config file on the NSX Manager as root user.
If in doubt, please open a Support Request to have a support engineer assist with this activity.

Log into the Manager shell as root user

1) Ensure the HTTP service is stopped. Note, due to this issue the service will likely already be stopped
    service proxy stop
2) Backup the server.xml config file
    cp /opt/vmware/proxy-tomcat/conf/server.xml /image
3) Edit /opt/vmware/proxy-tomcat/conf/server.xml
    This file must be editted so that there is only 1 Connector block of xml <Connector .... />
    - If the file contains 2 blocks of <Connector .../>, remove the first one.
    - If the file contains 3 blocks of <Connector .../>, remove the first two.
    - In all cases the last <Connector.../> block of xml is kept
4) In the remaining <Connector.../> block, delete the line that starts with "address=" e.g.
    address="1.2.3.4"
5) Save the file
6) Start the HTTP service
    service proxy start