Adding or redeploying a DLR from NSX fails with the error: “Deployment of edge gateway dlr-name failed. Failed while updating distributed router interfaces and dhcp relay on controller for NSX Edge edge-id”

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Adding or redeploying a DLR from NSX fails with an error similar to the below:
  
  Deployment of edge gateway dlr-name failed.  Failed while updating distributed router interfaces and dhcp relay on controller for NSX Edge edge-id., error code 10026, (org.hibernate.PropertyAccessException: Null value was assigned to a property [class com.vmware.vshield.vsm.vdn.nvpcontroller.model.VdrInstance.global] of primitive type setter of com.vmware.vshield.vsm.vdn.nvpcontroller.model.VdrInstance.global)
   

• Unable to access or ping networks behind DLR
• Ping from ESG internal interface to DLR uplink Interface fails
• Command output for net-vdr --instance -l on the ESXi host shows number of LIFs and routes as 0:

VDR Instance Information

---------------------------
Vdr Name: edge-628
Vdr Id: 0x000493f5
Number of Lifs: 0
Number of Routes: 0
Number of Hold Pkts: 0
Number of Neighbors: 0
State: Enabled
Controller IP: 192.168.189.25
Control Plane IP: 192.168.190.59
Control Plane Active: Yes
Num unique nexthops: 0
Generation Number: 0
Edge Active: No
Pmac: 00:00:00:00:00:00
Multicast Routing: Disabled
 

• ESXi host and NSX Manager logs indicate stale VDR table entries in the NSX Manager DB
• Entries similar to the below will be observed in the ESXi host vmkernel.log:

2021-12-29T13:34:54.727Z cpu7:2102109)WARNING: vdrb: VdrCpProcessVdrMcastUpdateMessage:3137: CP:Mcast Update: Failed for VDR [I:0x493e5] status: Bad parameter
 

• Entries similar to the below will be observed in the ESXi host vmkwarning.log:

2021-12-29T13:34:46.867Z cpu7:2102109)WARNING: vdrb: VdrCpProcessLifUpdateMessage:1348: CP:[I:0x493f4] Lif Update: Not able to find Active connection
 

• Entries similar to the below will be observed in the NSX Manageer vsm.log:

2022-01-24 08:59:52.833 GMT  INFO messagingTaskExecutor-9 VdrDvsInfoHandler:27 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing VdrDvsInfoRequest from: host-4390

Caused by: java.lang.IllegalArgumentException: Can not set boolean field com.vmware.vshield.vsm.vdn.nvpcontroller.model.VdrInstance.global to null value

Caused by: javax.persistence.PersistenceException: org.hibernate.PropertyAccessException: Null value was assigned to a property [class com.vmware.vshield.vsm.vdn.nvpcontroller.model.VdrInstance.global] of primitive type setter of com.vmware.vshield.vsm.vdn.nvpcontroller.model.VdrInstance.global

2022-01-24 09:00:00.134 GMT INFO TaskFrameworkExecutor-24 MissingResourcesHousekeeper:44 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Housekeeping: Started checking for stale Vdr Instances in DB
2022-01-24 09:00:00.158 GMT WARN TaskFrameworkExecutor-24 SqlExceptionHelper:129 - SQL Error: 0, SQLState: 23503
2022-01-24 09:00:00.159 GMT ERROR TaskFrameworkExecutor-24 SqlExceptionHelper:131 - ERROR: update or delete on table "vnvp_vdr_instance" violates foreign key constraint "vnvp_vdr_instance_backing_vdr_instance_fkey" on table "vnvp_vdr_instance_backing"
Detail: Key (vdr_id)=(300014) is still referenced from table "vnvp_vdr_instance_backing"

 

Environment

VMware NSX Data Center for vSphere 6.4.x

Cause

Stale VDR table entries are not cleaned up due to a housekeeping task error.  This precedes the symptoms and log entries detailed above.

Resolution

This issue is resolved in NSX for vSphere 6.4.14

Workaround:
Please raise a support request should you encounter this issue.