Application deployment failing intermittently after NSX Manager to Policy Mode promotion

Products

• VMware NSX
• VMware Tanzu Application Service

Issue/Introduction

• When you push an application, the application fails to stage due to an error when creating the container, and you see the error message in the application logs:

  Cell <cell_UUID> failed to create container for instance <UUID>: external networker up: exit status 1

• You are using NSX Container Plugin Tile and Tanzu Application Service.
• Failures started immediately following an NSX Manager-to-Policy promotion.
• You may observe the following errors in the nsx-node-agent/nsx-node-agent.stdout.log:

  [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="ERROR" errorCode="NCP01004"] nsx_ujo.agent.cni_watcher Unable to retrieve network info for container <contained-id>, network interface for it will not be configured
  [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="ERROR" errorCode="NCP01020"] nsx_ujo.agent.cni_watcher_lin Failed to delete veth pair <UUID> for container <UUID>, with err: (19, 'No such device').

• Using Traceflow shows Dropped by NEIGH by the Tier-1 DR of the Edge when using a source VM and the destination is the Diego cell IP.
• Using get neighbor in the T1-DR shows Incomp 00:00:00:00:00:00 for the affected Diego cell IP.
• You will not see the MAC in the MAC table for the segment the Diego cell is connected to:

  edge> get logical-switch <logical-switch_UUID> mac-address-table
      Interface   : <interface_UUID>
      IP          : <Diego_cell_IP>
      MAC         : 00:00:00:00:00:00
      State       : incomp
      Timeout     : 1

• Performing a vMotion of the Diego cell VM does not resolve the issue.

Environment

VMware NSX 4.2.4 and lower
VMware Tanzu Application Service

Cause

During an NSX Manager-to-Policy (MP2P) promotion, stale ports that existed prior to the promotion may be incorrectly associated as the active port for containers due to newer timestamps, causing network connectivity issues.

Resolution

This is a known issue impacting VMware NSX. We recommend you subscribe to this article to be updated on fix status.
Prior to a Manager to Policy promotion, use the scripted cleanup solution for stale ports to identify and clean any stale ports. In a future NSX version this check will be included in the MP2P promotion workflow.

Workaround:
Two methods to restore connectivity and resolve the application deployment failures, perform one of the following actions:

1. Remove the stale ports using the cleanup script referenced in Scripted cleanup of stale logical ports on NSX segments,
2. Restart nsx-node-agent on the affected Diego cells:
   sudo monit restart nsx-node-agent

or
   
   
3.  Recreate the Diego Cell.

Additional Information

Scripted cleanup of stale logical ports on NSX segments
Steps to remove stale logical-port(s) in NSX-T after attempted delete
NCP fails to stage container with external networker up error
Alarm for nsx-node-agent health status