Symptoms:
You can see error like " 2016-11-22T10:09:02+00:00 vse-1-0 ipsec[10453]: [78b0c80d-2680-4f84-82e1-d361e14fa5bb]: [authpriv.warning] packet from 87.245.122.246:500: received Vendor ID payload [XAUTH]
2016-11-22T10:09:02+00:00 vse-1-0 ipsec[10453]: [78b0c80d-2680-4f84-82e1-d361e14fa5bb]: [authpriv.warning] packet from 87.245.122.246:500: received Vendor ID payload [Dead Peer Detection]
2016-11-22T10:09:02+00:00 vse-1-0 ipsec[10453]: [78b0c80d-2680-4f84-82e1-d361e14fa5bb]: [authpriv.warning] packet from 87.245.122.246:500: ignoring Vendor ID payload [FRAGMENTATION 80000000]
2016-11-22T10:09:02+00:00 vse-1-0 ipsec[10453]: [78b0c80d-2680-4f84-82e1-d361e14fa5bb]: [authpriv.warning] packet from 87.245.122.246:500: received Vendor ID payload [RFC 3947] method set to=115
2016-11-22T10:09:02+00:00 vse-1-0 ipsec[10453]: [78b0c80d-2680-4f84-82e1-d361e14fa5bb]: [authpriv.warning] packet from 87.245.122.246:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02_n] meth=106, but already using method 115
2016-11-22T10:09:02+00:00 vse-1-0 ipsec[10453]: [78b0c80d-2680-4f84-82e1-d361e14fa5bb]: [authpriv.warning] packet from 87.245.122.246:500: initial Main Mode message received on 217.148.12.84:500 but no connection has been authorized with policy=PSK+XAUTH"
According to logs, Peer is initiating IKE transaction and in first message advertising PSK + XAUTH. As Edge is not configured to support XAUTH, it fails in first message.
When Edge is configured with Peer IP address, Edge can initiate IKE transaction in which it will not advertise XAUTH support and hence rest of the IKE transactions works as IKE will not prompt for authentication after IKE SA is established.
This issue occurs because NSX Edge receives the proposal for authentication with multiple methods like PSK XAUTH Certificate on Main Mode.
If the remote peer sends more than one in same time then the Edge cannot handle this proposal now edge supports PSK, XAUTH, Certificate authentication.
To resolve the issues: