When having communication problems that might be related to IPSec there are tests which can be executed to help you determine if IPSec is working correctly. Here are some excerpts from a Microsoft KB article (How To Configure IPSec Tunneling in Windows Server 2003) that will help you to test and troubleshoot IPSec problems. The article is also a useful reference for configuring new IPSec connections.
There are three tests you can use to determine whether your IPSec is working correctly:
TEST YOUR IPSec TUNNEL
You can initiate the tunnel by pinging from a computer on NetA to a computer on NetB (or from NetB to NetA). If you created the filters correctly and assigned the correct policy, the two gateways establish an IPSec tunnel so they can send the ICMP traffic from the ping command in an encrypted format. Even if the ping command works, verify that the ICMP traffic was sent in an encrypted format from gateway to gateway. You can use the following tools to do this.
ENABLE AUDITING FOR LOGON EVENTS AND OBJECT ACCESS
This logs events in the security log, tells you if IKE security association negotiation was tried, and whether it was successful or not.
To Enable Auditing:
CHECK THE IP SECURITY MONITOR
The IP Security Monitor console shows IPSec statistics and active security associations (SA). After you try to establish the tunnel by using the ping command, you can see if an SA was created (if the tunnel creation is successful, an SA is displayed). If the ping command is successful but there is no SA, the ICMP traffic was not protected by IPSec. If you see a "soft association" that did not previously exist, then IPSec agreed to allow this traffic to go "on the clear" (without encryption).
Note: In Microsoft Windows XP and the Windows Server 2003 family, IP Security Monitor is implemented as a Microsoft Management Console (MMC) console.
To add the IP Security Monitor snap-in, follow these steps: