Outlook fails to start when VPN active due to invalid egress IP address

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Users accessing WSS using WSS agents on Windows and MacOS

PFMS configured where WSS agent user pick up roaming PAC file

Office 365 restrictions setup such that conditional access only allowed if the client-ip matches corporate public IP address range or the Corporate VPN egress IP. WSS is not trusted because of shared IP addresses.

Most users have no issues accessing Outlook from a host running the WSS agent but a small subset of users coming from one specific IP egress range cannot connect even though the VPN is up and running

Outlook fails to start and it would appear that authentication is failing - typically because the IP address range is incorrect

Environment

VPN Client

O365 restrictions based on IP addresses

WSS Agent

Cause

PFMS PAC was created and assigned to location; then PAC file was removed but location was not resulting in the default PAC file being sent down to the user instead of roaming PAC file. This default PAC file location was sending traffic to proxy.threatpulse.com:8080 and being blocked upstream. When working, the O365 traffic should be sent through an internal proxy which is bypassed from going into WSS.

Location has 'keep agents active at this location' flag enabled to avoid going passive

Resolution

Remove the location so that users coming in from behind this location inherit the expected roaming PAC file.

Additional Information

When WSS agent sends traffic into WSS via the tunnel, it typically sends it to the Web server destination IP address. When troubleshooting this issue, we could see traffic going to the 199.19.250.205 (ep.threatpulse.net) Ip address. This typically means that the agent detected traffic destined for another proxy.

The rehydrated debug files that technical support has access to can confirm this, but another way of verifying it is to look at DNS requests on the public interface to see if there were any proxy server IP addresses being resolved. In our case, we could see a reference to proxy.threatpulse.net which should never be seen with the WSS agent.

From this point, we needed to understand where the request for this host came from, and looked at the PAC file in more details. Taking the PFMS Global PAC file URL pushed to the host and using curl to download it, we could see the following reference at the end of the PAC file

$ curl -vk https://pfms.wss.symantec.com/v1/pac?tenant=xxxxxxxxxxx
* STATE: INIT => CONNECT handle 0x80008f038; line 1835 (connection #-5000)
* Added connection 0. The cache now contains 1 members

< via: 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
function wssBypass(url, host) {
        return (
                shExpMatch(host, '*.oapp.com') || shExpMatch(host, 'oapp.com') ||
                shExpMatch(host, '*.jodo.es') || shExpMatch(host, 'jodo.es') ||
:
                isInNet(host, "17.7.94.01", "255.255.255.255")
        );
}
function FindProxyForURL(url, host)
{
    if (wssBypass(url, host)) return "DIRECT";

    if (isPlainHostName(host))
        return "DIRECT";

    if (url.substring(0, 4) == "ftp:")
        return "DIRECT";

    if (isInNet(host, "10.0.0.0", "255.0.0.0")
        || isInNet(host, "172.16.0.0",  "255.240.0.0")
        || isInNet(host, "192.168.0.0", "255.255.0.0")
        || isInNet(host, "169.254.0.0", "255.255.0.0")
        || isInNet(host, "127.0.0.0", "255.255.255.0"))
        return "DIRECT";

  return "PROXY proxy.threatpulse.net:8080;DIRECT";
}

This PAC file was the default PAC file assigned to anyone that does not match a PFMS condition, and not the PAC file that was assigned to roaming users (which sends 0365 traffic to a local proxy)

Looking at the PFMS logs, we could track the fact that a PFMS file has been created and deleted, but the location it had been assigned to was not deleted. This resulted in users from that location getting assigned the default PAC file and not the roaming user PAC file that previously worked. Here are the sequence of events we found from logs.

- Create a new location for your IP address - 2022-04-20 18:11:50.669 BST
- Copied prod PAC file to new PAC file for testing 
- Assigned your location to new PAC file - 2022-04-20 18:12:25.812 BST
- Deleted it - PAC file and its location association was deleted on 2022-04-20 18:34:18.948 BST ….
- but the location was not deleted until next day at 2022-04-21 14:31:16.265 BST