Loss of connectivity on all IPSec sessions on an NSX Edge

Products

• VMware NSX

Issue/Introduction

• On the active Edge the IKE CLI command may timeout.

edge>  get ipsecvpn session
[Timestamp]
% An unexpected error occurred: IKED CLI Timeout

• NSX Edge logs shows a recent state change (such as a failover) on the T1/T0 where IPSec is active.

/var/log/nsx-event.log
[Timestamp] NSX 1 FABRIC [nsx@6876 comp="nsx-edge" subcomp="nsxa" s2comp="ha-cluster" level="WARN" eventId="vmwNSXClusterFailoverStatus"] {"event_state":1,"event_external_reason":"Service router switches over from Active to Standby. rank0 serivce-router fails back","event_src_comp_id":"[UUID]","event_sources":{"id":"[UUID]","router_id":"[UUID]"}}
[Timestamp] NSX 1 FABRIC [nsx@6876 comp="nsx-edge" subcomp="nsxa" s2comp="ha-cluster" level="INFO" eventId="vmwNSXClusterFailoverStatus"] {"event_state":0,"event_external_reason":"Service router switches over from Standby to Active. ","event_src_comp_id":"[UUID]","event_sources":{"id":"[UUID]","router_id":"[UUID]"}}

• The same IKE message ID is used before and after the Edge failover. The reused mID can be seen in the Edge syslogs.

    /var/log/syslog

Before failover
[Timestamp]  NSX 1505413 VPN [nsx@6876 comp="nsx-edge" subcomp="iked" s2comp="ike-stack" level="INFO"] IKEv2 packet S([IP]:500 -> [IP]:500): mID=[ID], ([ID], [ID])(retransmit count=1)
...
After failover:
[Timestamp]  NSX 1505413 VPN [nsx@6876 comp="nsx-edge" subcomp="iked" s2comp="ike-stack" level="INFO"] IKEv2 packet S([IP]:500 -> [IP]:500): mID=[ID], ([ID], [ID])(retransmit count=7)

Environment

Vmware NSX-T Data Center 3.2.x
VMware NSX 4.x

Cause

Intermittently, post failover of an Edge, IKE packet initiated from a new active node uses the previously used ID. As a result, the peer sends the previous response though request is of a
different exchange type.

Resolution

This issue is resolved in VMware NSX 4.1.1 available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

Place Edge into maintenance mode and reboot