IPSec Tunnel Failure to WSS - Phase 1 IKE negotiation fails (loop or AUTH_FAILED)
search cancel

IPSec Tunnel Failure to WSS - Phase 1 IKE negotiation fails (loop or AUTH_FAILED)

book

Article ID: 407438

calendar_today

Updated On:

Products

Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

The customer just opened new site and the tunnel to Cloud Secure Web Gateway can not be established. Logs are showing Phase1 authentication failures

Environment

Environment Overview

  • Customer Egress IP: x.x.x.x
  • WSS POP IP (Abu Dhabi): 168.149.175.164
    Ref.: Cloud SWG (formerly WSS) Ingress and Egress IP addresses
  • Tunnel Type: IPSec (from Fortinet firewall)
  • Authentication: Pre-Shared Key (PSK)
  • Failure Symptom: Phase 1 IKE negotiation fails (loop or AUTH_FAILED)

Cause

Findings

1. Tunnel Configuration Is Reaching the POP

  • Traffic is reaching the WSS POP (168.149.175.164), as indicated by:

    • Phase 1 negotiation logs appearing in Fortinet. 

      "Logs are showing Phase1 authentication failures."
      "We switched from IKEv2 to IKEv1 and back but that did not make any change."
      "We tested local ID being (email address, blank, Egress IP address) but when Egress IP was there the phase1 was repeating in the loop."
      "When the field was either blank or an email we do see authentication failed."

      "Traffic is inferred to be reaching the WSS POP based on Fortinet logs showing Phase 1 authentication failures and looping behavior - as you have described in the case description."
      These symptoms indicate that Phase 1 negotiation is being attempted and partially processed, not completely blocked at the network layer.

    • Looping behavior or authentication errors - not silence/timeouts

2. Phase 1 Fails with Authentication Errors

  • Repeated Phase 1 failures, especially:

    • AUTHENTICATION_FAILED

    • HASH N(AUTH_FAILED)

  • Testing different Local ID values (email, blank, Egress IP) affected behavior but did not resolve it

Root Cause: Local ID (IDi) Does Not Match Registered Egress IP

When the Local ID (IDi) presented during IKE Phase 1 does not match the IP registered in the WSS portal, the WSS POP will reject the connection during authentication.

This is not described verbatim in the IPSec setup TechDocs, but it is clearly explained in the following official source:

[1] KB Article: “HASH N(AUTH_FAILED)” error when trying to establish VPN tunnel

Ref.: HASH N(AUTH_FAILED)

“Set the local ID to the given public egress IP. The IP should be the same as added in the Cloud SWG portal (…) under Connectivity > Location > Access Method Firewall VPN.”

This KB is the authoritative source confirming that the Local ID must match the registered egress IP.

[2] WSS Connectivity TechDoc – IPSec Setup Requirements

Ref.: Troubleshoot IPsec/VPN/Firewall Connections

Covers general IPSec setup guidance:

  • Supported ports/protocols

  • Phase 1/2 encryption proposals

  • NAT-T and IKEv1/v2 compatibility

[3] Additional Reference: 

Ref.: Broadcom KB: IPSec Tunnel Setup Failure

Includes general troubleshooting for:

  • IKE version mismatches

  • NAT-T issues

  • PSK verification

Resolution

Recommended Actions to Resolve the Issue

Step 1: Set Local ID Explicitly to x.x.x.x

In the Fortinet IPSec Phase 1 config:

  • Set Local ID / Local Identifier (IDi) to x.x.x.x

  • This must match what is configured in the WSS Portal under:

    Connectivity > Locations > [Your Location] > Firewall VPN > Registered IP Address

 

In Fortigate CLI (Please have your Fortigate Tech. Suport to validate this):

config vpn ipsec phase1-interface

edit <tunnel_name>

set localid x.x.x.x

next

end

Step 2: Verify Pre-Shared Key

  • Ensure the PSK used in Fortinet matches exactly the one in WSS portal (no extra spaces or invisible characters).

Step 3: Ensure Compatibility with Broadcom’s IPSec Requirements

Recommended Phase 1/IKE settings:

Parameter Recommended Value
IKE Version IKEv1 (Main Mode) or IKEv2
Encryption AES-256
Integrity/Hash SHA-256
DH Group Group 14 (2048-bit)
NAT-T Enabled (if NAT present)
Lifetime 28800 seconds

Also ensure outbound UDP 500 and 4500 are allowed.

Optional Verification: Debug IKE Phase 1 on FortiGate

Purpose

To confirm:

  • Whether IKE Phase 1 messages are being sent/received

  • If local-id or psk mismatches are causing AUTH_FAILED

  • If the WSS POP is reachable and replying

Step-by-Step Commands (Ensure to work with your Fortigate Tech. Support, for these)

Only run these during a scheduled test window, as debug logging is verbose and may affect performance in high-traffic environments.

1. Disable all previous debugs

diagnose debug resetdiagnose debug disable

2. Enable IKE debug for Phase 1 and 2

diagnose vpn ike log-filter cleardiagnose vpn ike log-filter dst-addr4 168.149.175.164diagnose vpn ike log-filter interface <your_wan_interface_name>diagnose debug application ike -1diagnose debug enable

Replace <your_wan_interface_name> with the real WAN or tunnel interface used to reach WSS.

3. Initiate the tunnel (or wait for it to auto-attempt)

execute vpn tunnel up name <tunnel_name>
 
4. Look for Key Outputs

Look for the following in the debug output:

  • proposal chosen → good indication Phase 1 negotiation started

  • sending XAUTH request → seen in hybrid mode (not needed here)

  • peer ID is ... and local ID is ... → shows the actual IDs exchanged

  • HASH N(AUTH_FAILED) or AUTHENTICATION_FAILED → key sign of Local ID or PSK mismatch

5. Disable debug after capture

diagnose debug disable

Optionally, you can also run:

diagnose debug reset
 
Optional: Save logs for review

You can also redirect debug output to a file for offline analysis if supported by your device.

What to Look For

Symptom Likely Cause
No response after sending... WSS POP unreachable (routing/firewall issue)
AUTHENTICATION_FAILED Incorrect Local ID or PSK
proposal mismatch Wrong IKE proposal settings
NAT detection and no reply Try enabling NAT-T

 

Fortinet KB: Troubleshooting Tip: IPsec Tunnel debugging IKE

Based on all the findings so far, it is reasonable to conclude that the issue is on the Fortinet side (customer-side configuration), and not with the WSS (Cloud SWG).

Summary

Why it's not a WSS issue:

  1. WSS POP (168.149.175.164) is known to be operational and used by other customers successfully.

  2. You’ve confirmed the correct destination IP is configured (Abu Dhabi POP).

  3. The failure is during Phase 1 (authentication) - WSS will only reject Phase 1 if:

    • The Local ID (IDi) doesn’t match the registered egress IP, or

    • The PSK is incorrect

  4. From your description:

    • When Local ID was set to x.x.x.x (correct egress IP), Fortinet got stuck in a repeating loop - typical of Phase 1 retrying due to failed auth, not network unreachable.

    • When Local ID was blank or set to an email, logs showed AUTHENTICATION_FAILED, which aligns with Broadcom rejecting the IDi as invalid.

In all cases, the tunnel reaches the WSS POP but is rejected due to mismatched credentials or ID, which means the WSS side is reachable and responsive, not faulty.

Powered by Wolken Software