NSX-T Edge cannot enter/exit Maintenance Mode.

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• The Edge cannot enter or exit Maintenance Mode.
• In the Edge syslog file (var/log/syslog), you may see:

<179>1 2020-02-18T16:29:49.873725+00:00 edgetn01 NSX 29783 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="upgrade-bundle" level="ERROR"] Error sending request to 'http://127.0.0.1:7441/api/v1/node/maintenance-mode': <HTTPError 500: 'Internal Server Error'>

• in the Edge api access log (/var/log/nvpapi/api_access.log), you may see the time to execute this command is more than few seconds (In the below example, 4722.33 seconds).

2020-02-18T16:29:49.872Z api_access.80 INFO 127.0.0.1 - - [2020-04-11 12:26:49] 'PUT /api/v1/node/maintenance-mode --- New value: {"maintenance_mode": true}' 500 161 "" "Python-urllib/3.5" 4722.339133

Depending on the situation, '{"maintenance_mode": true}' could also be '{"maintenance_mode": false}'

• Accessing the Edge TN in SSH (with Root user) and run "ps -ef".
• The below command should be running for only a few seconds, less than 10 seconds in most circumstances.

14548 1779 33 www-data 00:00:00 0.0 0.0 3564 25040 /usr/bin/nsx-appctl -t /var/run/vmware/nsx-opsagent/opsagent-cli edgesvc/cli/change mmode

​​​

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.5.x

Cause

The process to enter/exit maintenant mode on the Edge is stuck in an invalid state.

Resolution

This issue is fixed in NSX-T for Data Center versions: 3.0: https://customerconnect.vmware.com/downloads/info/slug/networking_security/vmware_nsx_t_data_center/3_x
2.5.2: https://customerconnect.vmware.com/downloads/info/slug/networking_security/vmware_nsx_t_data_center/2_x

Workaround:
To workaround this issue, restart the impacted Edge TN.