Pod deployment fails: Error adding container to network nsx-cni: Failed to connect to nsx_node_agent

Products

• VMware NSX

Issue/Introduction

• Issue can affect all VMs including Worker and Master nodes.
• The following alarms may trigger. 
  • NCP Plugin Down 
  • Node Agents Down

• Within NSX UI Inventory > Containers > Clusters > Network Status = Down.
• Network Status Info icon may return.

NCP00010: Failed to get node vif or TN ID for node
NCP00143: Failed to create segment port

• NSX Container Plugin Operator logs show the following (this logging only applies to Openshift deployments)

         oc logs <operator> -n <namespace>

{"level":"info","ts”:”[TIMESTAMP]”,”logger":"controller_node","msg":"Skip reconcile: node was processed","Request.Namespace":"","Request.Name”:”[HOSTNAME]”}
{"level":"info","ts”:”[TIMESTAMP]”,”logger":"controller_node","msg":"Skip reconcile: node was processed","Request.Namespace":"","Request.Name”:”[HOSTNAME]”}

• Logical port for affected VM(s) is missing the following tags.

scope='ncp/node_name', tag=‘[NODE NAME]’
scope='ncp/cluster', tag=‘[CLUSTER NAME]’

• Port output where tags are missing.

GET NSX-Manager/nsxapi/api/v1/logical-ports
{
        "_create_time": [TIMESTAMP],
        "_create_user": "system",
        "_last_modified_time": [TIMESTAMP],
        "_last_modified_user": "system",
        "_protection": "NOT_PROTECTED",
        "_revision": 2,
        "_system_owned": false,
        "address_bindings": [],
        "admin_state": "UP",
        "attachment": {
          "attachment_type": "VIF",
          "id": "[UUID]"
        },
        "display_name": "[VM NAME]@[PORT UUID]", <<<< Affected VM name & port
        "id": "[UUID]",
        "ignore_address_bindings": [],
        "internal_id": "[UUID]",
        "logical_switch_id": "[UUID]",
        "resource_type": "LogicalPort",
…
        "tags": [ 
          {
            "scope": "policyPath",
            "tag": "/infra/segments/[SEGMENT NAME]/ports/default:[UUID]"
         <<<<Tags should be listed here>>>>>
          }
        ]
      },

 

• Within vCenter the Tasks/Events shows that the problematic VM was recently migrated between ESXi hosts.
• During this migration connectivity to the NSX Manager is interrupted.
• Interruption may have been caused by general network connectivity issues or instability on the NSX Manager cluster similar to NSX Manager Cluster Degraded

 

 

Environment

VMware NSX
VMware NSX-T Data Center

Cause

During VM migration detach and attach requests are sent to the NSX Manager. Due to instability either on the NSX Manager or with connectivity between ESXi host and NSX Manager the manager may only receive a detach request. As result, a new port is created for the migrated VM which doesn't contain the specific tags.

Resolution

There's currently no resolution to this issue.

Workaround:
Manually re add the tags via the NSX User Interface - Add Tags