NSX Edge cannot enter/exit Maintenance Mode.

Products

• VMware NSX

Issue/Introduction

• The Edge cannot enter or exit Maintenance Mode.
• In the Edge syslog file (var/log/syslog), you may see:
  
  

  <179>1 2020-02-18T16:29:49.873725+00:00 edgetn01 NSX 29783 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="upgrade-bundle" level="ERROR"] Error sending request to 'http://127.0.0.1:7441/api/v1/node/maintenance-mode': <HTTPError 500: 'Internal Server Error'>

• in the Edge api access log (/var/log/nvpapi/api_access.log), you may see the time to execute this command is more than few seconds (In the below example, 4722.33 seconds).
  
  

  2020-02-18T16:29:49.872Z api_access.80 INFO 127.0.0.1 - - [2020-04-11 12:26:49] 'PUT /api/v1/node/maintenance-mode --- New value: {"maintenance_mode": true}' 500 161 "" "Python-urllib/3.5" 4722.339133

  Depending on the situation, '{"maintenance_mode": true}' could also be '{"maintenance_mode": false}'

• Accessing the Edge TN in SSH (with Root user) and run "ps -ef".
• The below command should be running for only a few seconds, less than 10 seconds in most circumstances.
  

  14548 1779 33 www-data 00:00:00 0.0 0.0 3564 25040 /usr/bin/nsx-appctl -t /var/run/vmware/nsx-opsagent/opsagent-cli edgesvc/cli/change mmode

• The error could appear during an NSX edge upgrade and will cause the upgrade to fail.

Environment

VMware NSX

VMware NSX-T Data Center

Cause

The process to enter/exit maintenance mode on the Edge is stuck in an invalid state.

Resolution

This issue is resolved in VMware NSX 3.0 and later, available at Broadcom downloads.

If you are having difficulty finding and downloading software, review the Download Broadcom products and software KB.

Workaround:

To workaround this issue, restart the impacted Edge TN.