VMs cannot connect to Logical Switches on one or more ESXi hosts

Products

• VMware NSX

Issue/Introduction

Symptoms:

• VMs cannot connect to Logical Switches on one or more ESXi hosts.
• ESXi host logs (nsx-opsagent.log) display message(s) similar to:

2019-10-28T13:11:03Z nsx-opsagent[74209]: [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="74344" level="ERROR" errorCode="MPA42046"] [SyncVnicAttachment] unable to sync attachment [opId:[sync-attach-2] op:[SYNC_ATTACH_PORT(1001)] vif:[d28c2830-64b7-49bd-XXXX-XXXX7c13141e] ls:[0a7f47b8-9a8b-4943-XXXX-XXXX93a08742] vmx:[/vmfs/volumes/5c1a0eca-3315d27c-6e48-XXXXXXXXXX/f7e8bf42-fc62-4841-85b6-XXXXXXXXX/f7e8bf42-fc62-4841-XXXX-3740b22XXXX.vmx] lp:[]]: Cannot operate in maintenance mode

• The ESXi host is in NSX-T Maintenance mode. Use the following REST API to verify.

GET /api/v1/transport-nodes/{TRANSPORT_NODE_ID}/state

Example of output:
{
  "transport_node_id" : "23fbf29b-708f-4f8d-87f1-d5ede0fb727d",
  "host_switch_states" : [ {
    "host_switch_id" : "1e f7 08 64 be 1e 48 fe-80 9c d4 25 ed 2e a7 31",
    "host_switch_name" : "NVDS",
    "endpoints" : [ {
      "device_name" : "vmk10",
      "ip" : "192.168.141.12",
      "default_gateway" : "192.168.141.254",
      "subnet_mask" : "255.255.255.0",
      "label" : 65541
    } ],
    "transport_zone_ids" : [ "b54a3810-5140-447d-b396-a318f53f774a", "ec3cb584-357d-4d62-94d3-e2c525d411f4" ]
  } ],
  "maintenance_mode_state" : "ENABLED",  <<--------
  "node_deployment_state" : {
    "state" : "success",
    "details" : [ ]
  },
  "state" : "success"
}

Environment

VMware NSX-T Data Center 2.x
VMware NSX-T Data Center
VMware NSX-T

Cause

The issue may occur after NSX-T is upgraded on the ESXi host and the file /etc/vmware/nsxa/host_config_mmode.bin is not emptied causing the ESXi host to be still considered in NSX-T Maintenance Mode.

Resolution

This issue is resolved in NSX-T 2.5.0.

Workaround:
As a workaround the following steps can be applied to clear the NSX-T Maintenance Mode status:

1. SSH to the ESXi host.

2. Rename the file /etc/vmware/nsxa/host_config_mmode.bin
#mv /etc/vmware/nsxa/host_config_mmode.bin /etc/vmware/nsxa/host_config_mmode.bin.bck

3. Restart the nsx-opsagent service:
#/etc/init.d/nsx-opsagent restart

Issue also observed in NSX 4.1.1 (Same Resolution as above)