After Disabling/Removing 3'rd Party Service Insertion/Deployment, disconnected NICs are observed for any VM that is powered off/on or vMotioned.

Products

• VMware NSX

Issue/Introduction

This document is being created to help understand why and under what conditions these disconnected NIC's are seen.  The workaround and fix for this problem is also outlined here.

Symptoms:

• The customer observes disconnected NICs for any VM's that have been powered off/on of vMotion'd.
• If the customer creates a new VM, vMotions existing VM, power off/on existing VM, the vmxnet3 adapter on the VM does NOT start. Device status shows "This device cannot start. (code 10)."
• This behavior is seen after a 3'rd party Service Deployment/Insertion has been removed from the environment. 

Environment

VMware NSX-T

Cause

This is due to the slot 12 filter on the VM vnic (used by North-South and East-West Service Insertion) not being removed correctly after the Service deployment has been removed/deleted.  NSX-T owned Service Plane Forwarding port (SPF port) is then sometimes removed incorrectly during a subsequent DVS monitor cycle by vCenter, as they are not recognized by vCenter.

Resolution

The fix for this issue is actually included in ESX 7.0.2 P03.

Workaround:

STEP1:Find out if E-W and N-S Service Insertion are enabled.

GET /policy/api/v1/infra/settings/service-insertion/security/status/

 

STEP2: Disable both E-W and N-S SI by placing false value 

PUT /policy/api/v1/infra/settings/service-insertion/security/status/

{

 "north_south_enabled": false, >>>>>>>>>>>>>>>>>>>>>>>>>
 "east_west_enabled": false,   >>>>>>>>>>>>>>>>>>>>>>>>>
 "resource_type": "PolicySIStatusConfiguration",
 "id": "status",
 "display_name": "status",
 "path": "/infra/settings/service-insertion/security/status",
 "relative_path": "status",
 "parent_path": "/infra",
 "unique_id": "########-####-####-####-##########70",
 "marked_for_delete": false,
 "overridden": false,
 "_create_user": "system",
 "_create_time": 0,
 "_last_modified_user": "system",
 "_last_modified_time": 1613654641196,
 "_system_owned": false,
 "_protection": "NOT_PROTECTED",
 "_revision": 1
}

 

STEP3: Run below API to validate the logical-switches with .gvm properties

GET /api/v1/logical-switches?diagnostic=true

 

Finally  run  below commands to remove any .gvm properties:

  net-dvs -u com.vmware.nsx.spf.gvm -p <portid> <dvs-name>
  net-dvs -u com.vmware.nsx.spf.enabled -p <portid> <dvs-name>
  net-dvs -u com.vmware.port.extraConfig.serviceInsertion.gvm -p <portID> <dvs-name>

NOTE : This last commands are executed from ESXi host where affected VMs are running.