VCD created route based IPSEC VPN reported as down with a 'Peer not reachable' error due to lost route redistribution configuration

Products

• VMware NSX
• VMware Cloud Director

Issue/Introduction

• IPsec VPN tunnels show a status of Down or Peer not reachable. The IPSEC tunnel was created using VCD (vCloud Director).
• When a T0 router is created via vCD, then it will automatically configure a route redistribution rule on the T0, which is named 'SYSTEM-VCD-EDGE-SERVICES-REDISTRIBUTION'.
• In the NSX Manager UI, navigate to Networking > Connectivity > Tier-0 Gateways > [Gateway Name] > Route Redistribution. It is found that the 'IPSEC Local Endpoint' routes are no longer enabled for redistribution on any of the redistribution rules.
• Also, it is found that the 'SYSTEM-VCD-EDGE-SERVICES-REDISTRIBUTION' rule is missing, which is the rule that had the 'IPSec Local Endpoint' redistribution flag enabled.
• In the Edge /var/log/syslog there is a similar log message which indicates that a redistribution rule list update has been received with an action to delete:
  
  NSX 3006 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="rcpm" s2comp="nestdb" level="INFO"] Handle MONITOR message action: update, type 16: REDISTRIBUTION_RULE_LIST, update: 78 bytes, delete: 129 bytes
NSX 3006 ROUTING [nsx@6876 comp="nsx-edge" subcomp="rcpm" s2comp="rcpm-db" level="INFO"] Found RedistributionRuleList UUID(<UUID>) Updating


• That is followed by a log which confirms that the routing prefix for the 'Local Endpoint IP' has been deleted:
  
  NSX 3006 ROUTING [nsx@6876 comp="nsx-edge" subcomp="rcpm" s2comp="rcpm-nsxa" level="INFO"] Received prefix <LOCAL ENDPOINT IP>/32 with n_nexthops = 0 in table_id = 258 action = DELETE
NSX 1 ROUTING [nsx@6876 comp="nsx-edge" subcomp="nsxa" s2comp="routing" level="INFO"] FIB update 2 delete <LOCAL ENDPOINT IP>/32 delete

• Soon after that the remote VPN peer is marked as dead and the IPSEC tunnel is reported as DOWN:
  

  NSX 3044266 VPN [nsx@6876 comp="nsx-edge" subcomp="iked" s2comp="ike-stack" level="INFO"] IKE peer '<REMOTE PEER IP>' marked as Dead.
NSX 3044266 VPN [nsx@6876 comp="nsx-edge" subcomp="iked" s2comp="iked-event" level="WARN" eventId="vmwNSXVpnTunnelState"] {"event_state":0,"event_external_reason":"IPSec tunnel (563580390) status DOWN: 

• In the NSX Manager /var/log/syslog you might find a log similar to the following, which indicates that a PATCH API operation was run against the T0, at the time of the redistribution list update seen in the Edge logs. The IP address listed in the log is the source of the PATCH API command such as an automation workflow:
  
  [nsx@6876 audit="true" comp="nsx-manager" level="INFO" reqId="<REQUEST UUID>" splitId="XrbwBXsc" splitIndex="1 of 2" subcomp="manager" update="true" username="admin"] UserName="admin", Src="IP OF THE API COMMAND SOURCE>", ModuleName="PolicyConnectivity", Operation="PatchTier0", Operation status="success"

Environment

VMware NSX
VMware Cloud Director

Cause

• This issue is caused by the default behavior of the NSX Policy API during hierarchical PATCH requests. A hierarchical PATCH API request executed by an external client updates the LocaleServices configuration but does not include the 'SYSTEM-VCD-EDGE-SERVICES-REDISTRIBUTION' rule in its payload, causing the Hierarchical API framework to overwrite and delete the pre-existing system rule.
• When an external automation tool sends a PATCH request to a parent object (like a Tier-0 Gateway or its Locale Services), the NSX Manager interprets the provided payload as the complete desired state for that specific section. If the specific redistribution rules are omitted from the payload, the API overwrites the existing configuration with the partial data provided, effectively deleting the omitted rules.
• This commonly occurs when automation workflows are configured to update a single attribute (e.g., a description or a different service) without first performing a GET to include the existing state of system-generated rules like SYSTEM-VCD-EDGE-SERVICES-REDISTRIBUTION.

Resolution

• As a workaround to restore connectivity, manually re-enable the missing redistribution rules in the NSX UI:
  1. Navigate to the affected Tier-0 Gateway.
  2. Edit the Route Redistribution configuration.
  3. Ensure the 'SYSTEM-VCD-EDGE-SERVICES-REDISTRIBUTION' rule is present and that the appropriate checkbox 'IPSec Local Endpoint' is selected.
     
     
• To prevent this issue in automation workflows:
  • Method 1 (Recommended): Implement a Read-Modify-Write workflow. Perform a GET call to retrieve the full object configuration, modify only the required field in the JSON, and then PATCH the entire payload back to the NSX Manager.
    • The payload should be formed by first doing a "GET https://<nsx-manager>/policy/api/v1/infra/tier-0s/<TIER-0-ID>/locale-services/default" and include the response from the GET API that will contain all the redistribution rules including 'SYSTEM-VCD-EDGE-SERVICES-REDISTRIBUTION' as expected.
  • Method 2: Explicitly use the Partial Patch feature by appending ?partial_patch=true to the API request URL. This instructs NSX to merge the payload with the existing configuration rather than overwriting it.

Additional Information

• KB 422518: Route advertisement rules deleted from Tier-1 Gateway after API PATCH update — Explains the generic behavior of PATCH/PUT API calls in NSX.
• NSX-T Data Center REST API Documentation: Partial Patch