Troubleshoot and Test IPSec Communication

Products

• Data Loss Prevention Network Monitor
• Data Loss Prevention Enforce

Issue/Introduction

When having communication problems that might be related to IPSec there are tests which can be executed to help you determine if IPSec is working correctly.  Here are some excerpts from a Microsoft KB article (How To Configure IPSec Tunneling in Windows Server 2003​) that will help you to test and troubleshoot IPSec problems. The article is also a useful reference for configuring new IPSec connections.

Resolution

There are three tests you can use to determine whether your IPSec is working correctly: 

• Test your IPSec tunnel 
• Enable auditing for logon events and object access 
• Check the IP security monitor

TEST YOUR IPSec TUNNEL

You can initiate the tunnel by pinging from a computer on NetA to a computer on NetB (or from NetB to NetA). If you created the filters correctly and assigned the correct policy, the two gateways establish an IPSec tunnel so they can send the ICMP traffic from the ping command in an encrypted format. Even if the ping command works, verify that the ICMP traffic was sent in an encrypted format from gateway to gateway. You can use the following tools to do this.


ENABLE AUDITING FOR LOGON EVENTS AND OBJECT ACCESS

This logs events in the security log, tells you if IKE security association negotiation was tried, and whether it was successful or not.

To Enable Auditing: 

1. Using the Group Policy MMC snap-in, expand Local Computer Policy, expand Computer Configuration, expand Windows Settings, expand Security Settings, expand Local Policies, and then click Audit Policy.
2. Enable Success and Failure auditing for Audit logon events and Audit object access. Note:  If the Windows Server 2003 gateway is a member of a domain and if you are using a domain policy for auditing, the domain policy overwrites your local policy. In this case, modify the domain policy.

CHECK THE IP SECURITY MONITOR

The IP Security Monitor console shows IPSec statistics and active security associations (SA). After you try to establish the tunnel by using the ping command, you can see if an SA was created (if the tunnel creation is successful, an SA is displayed). If the ping command is successful but there is no SA, the ICMP traffic was not protected by IPSec. If you see a "soft association" that did not previously exist, then IPSec agreed to allow this traffic to go "on the clear" (without encryption).

Note: In Microsoft Windows XP and the Windows Server 2003 family, IP Security Monitor is implemented as a Microsoft Management Console (MMC) console.

To add the IP Security Monitor snap-in, follow these steps:

1. Select Start, Run. 
2. Type MMC, click OK. 
3. Click File, Add/Remove Snap-in, click Add. 
4. Click IP Security Monitor, click Add. 
5. Click Close, click OK.