NSX-T Edge cannot enter/exit Maintenance Mode.

Products

• VMware NSX

Issue/Introduction

• The Edge cannot enter or exit Maintenance Mode.
• In the Edge syslog file (var/log/syslog), you may see:
  
  

  <179>1 2020-02-18T16:29:49.873725+00:00 edgetn01 NSX 29783 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="upgrade-bundle" level="ERROR"] Error sending request to 'http://127.0.0.1:7441/api/v1/node/maintenance-mode': <HTTPError 500: 'Internal Server Error'>

• in the Edge api access log (/var/log/nvpapi/api_access.log), you may see the time to execute this command is more than few seconds (In the below example, 4722.33 seconds).
  
  

  2020-02-18T16:29:49.872Z api_access.80 INFO 127.0.0.1 - - [2020-04-11 12:26:49] 'PUT /api/v1/node/maintenance-mode --- New value: {"maintenance_mode": true}' 500 161 "" "Python-urllib/3.5" 4722.339133

  Depending on the situation, '{"maintenance_mode": true}' could also be '{"maintenance_mode": false}'

• Accessing the Edge TN in SSH (with Root user) and run "ps -ef".
• The below command should be running for only a few seconds, less than 10 seconds in most circumstances.
  
  

  14548 1779 33 www-data 00:00:00 0.0 0.0 3564 25040 /usr/bin/nsx-appctl -t /var/run/vmware/nsx-opsagent/opsagent-cli edgesvc/cli/change mmode

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.5.x

Cause

The process to enter/exit maintenance mode on the Edge is stuck in an invalid state.

Resolution

This issue is fixed in NSX-T Data Center 3.0.

Workaround:
To workaround this issue, restart the impacted Edge TN.