Following an upgrade to NSX-T 3.2.0 or later, IKED service crashes seen with VPN tunnels going down

Products

• VMware NSX

Issue/Introduction

• NSX-T 3.2.0 or later is deployed. 
• Multiple IKED cores are seen post upgrade and do not recover.
• VPN flows get disrupted until the workaround is applied manually.
• Bypass Policy is configured in IPSEC service with either "Local Networks" or "Remote Networks" as empty.
• Running get ipsecvpn session summary on the Edge Node results in an output similar to the below:
  
  

  Thu May 30 2024 EDT 04:25:27.853
  
  Version  SID  Compliance Suite Type    Auth  Status        Local IP         Peer IP          Down Reason
  ----------------------------------------------------------------------------------------------------------------------------
  IKEv2    0    NONE             Policy  PSK   Down          <Local IP>       <Peer IP>        IKED waiting for RSS queue info from DP
  IKEv1    0    NONE             Policy  PSK   Down          <Local IP>       <Peer IP>        IKED waiting for RSS queue info from DP
  IKEv2    0    NONE             Policy  PSK   Down          <Local IP>       <Peer IP>        IKED waiting for RSS queue info from DP
  IKEv2    0    NONE             Policy  PSK   Down          <Local IP>       <Peer IP>        IKED waiting for RSS queue info from DP
  ----------------------------------------------------------------------------------------------------------------------------

• IKED core dumps will be visible in /var/dump:
  

  -rw-r--r--  1 root root 2.0M May 30 01:03 core.iked.#.gz

• Entries similar to the below will be visible in /var/log/kern.log:
  

  2024-05-30T05:03:47.852Z edge01 kernel - - - [ 6277.247167] grsec: Segmentation fault occurred at 0000000000000038 in /opt/vmware/nsx-edge/bin/iked[iked:28091] uid/euid:150/150 gid/egid:150/150, parent /opt/vmware/edge/ike/entrypoint.sh[entrypoint.sh:27973] uid/euid:150/150 gid/egid:150/150
  

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center
VMware NSX

Cause

Prior to NSX-T 3.2.0, any empty field in the Bypass Policy would get mapped to IP address as 0.0.0.0.

In NSX-T 3.2.0 and later, the mapping is missing, due to which the nestdb attributes for local/remote appear with NULL values, this leads to crashing while processing occurs in IKED, as the value is expected but is not present.

Resolution

This issue is resolved in VMware NSX 4.2.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

 

Workaround

Before Upgrade:

• If the Bypass Policy is currently configured without specifying local or remote networks (i.e., the fields are empty), update the configuration to include a wildcard IP address of 0.0.0.0/0 for both fields.
• Navigate to:
  Networking > VPN > VPN Services > [Select your VPN Service] > Expand the Bypass Policy section

After Upgrade: 

• If the NSX environment has already been upgraded and the issue persists, update the Bypass Policy settings by setting both the local and remote networks to 0.0.0.0/0 as described above.
• After updating the settings, restart the IKED service on the Edge node to apply the changes by running the following command from the Edge node’s root shell:

  # docker start service_iked

Additional Information