NSX-T error 5103 'logical router configuration realization error' and or 5105 'logical router FIB configuration realization error'

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX-T version 3.2.x.
• North South service insertion is used.
• In NSX-T Manager UI  you may see Tier-1 and Tier-0 Logical Router realization errors, Error Code = '5103', Error Message = 'logical router configuration realization error'.
• In NSX-T Manager UI  you may also see Tier-1 and Tier-0 Logical Router realization errors, Error Code = '5105', Error Message = 'logical router FIB configuration realization error'.
• This can lead to North/South dataplane impact, affect new and existing Logical Routers.
• In the NSX-T manager /var/log/syslog you may see error similar to:

023-03-24T10:25:16.026Z manager1.corp.local NSX 6446 - [nsx@6876 comp="nsx-controller" level="WARNING" subcomp="domain"] No SrClusterId for Layer3SiPolicy message managed_resource { display_name: "396fa74d-a77d-45f1-8548-b44732dea068"}service_insertion_policy { owner_id { c636581c-63c3-4e91-8abd-a5113b114c30#012 }}service_port_nhip { ipv4: 10.10.10.10}logical_router_id { 033cd93c-2c27-4eeb-b2fa-99d1980d48aa}

• In the NSX-T manager /var/log/cloudnet/nsx-ccp.log you may see error similar to: 

2023-03-27T15:47:20.962Z WARN Owl-worker-4 ExecutorTask 1542 - [nsx@6876 comp="nsx-controller" level="WARNING" subcomp="owl"] Failed to process dataUpdate for listener RedistributionAppImpl, error message: null, error stack:java.lang.NullPointerException

...

Environment

VMware NSX-T Data Center
VMware NSX-T

Cause

This is due to a Service Deployment having reference to a removed/deleted logical router.  
While attempting to publish a nestdb (internal DB) message, CCP (Central Control Plane) encountered a NPE (Null Pointer Exception) caused by the absence of the SrClusterId value. This resulted in the transaction being blocked.

Resolution

This is a known issue impacting NSX-T Data Center.

Workaround:

If you believe you have encountered this issue, please open a support request and refer to this KB.

Additional Information

Impact/Risks:

Logical routers may go in to an unknown state with realization errors.
North/South traffic may cease and East/West traffic may also be affected.