"Invalid or expired customer, Internet access blocked" error reported in WSS Agent with no user connectivity allowed

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Cloud SWG service used to protect internet traffic for on premise and roaming users.

WSS Agent users started reporting "Invalid or expired customer, Internet access blocked" error when connecting from the office network. All connectivity and internet access was blocked when in this state.

None of the WSS Agent users connecting from home had issues.

The above error correlated to a Cloud SWG configuration change made by the admin, where the office network egress IP was removed from the “Locations” field. The objective was to have office users go ACTIVE with the WSS Agent.

Re-adding the Egress IP back to the “Locations” in Cloud SWG Portal fixed the issue.

Environment

WSS Agent.

On premise and roaming users.

Cause

Office egress IP address GEOlocation settings not fully configured in maxmind.

Resolution

Two changes made:

1. Went back to Maxmind and added all GEO IP details to the egress IP address entry.
2. Pushed out Cloud SWG changes to ignore WSS Agent authentication requests that included incomplete GEOlocation parameters (country code exists but is blank).

Additional Information

WSS Agent logs (had no Symdiag unfortunately) confirmed that CTC communication returned Cloud SWG connection list IP addresses as expected, and that the agent did try and connect to these before erroring out:

[08:05:52]: CTC Response: ACTIVE(GEOIP)  egress:203.0.113.1  GCHZU-148.64.11.170  GITMI-148.64.10.164  GDEFR-199.247.40.164  
[08:05:53]: Attempting to connect to GCHZU via UDP
[08:06:15]: CA Tunnel#0(non-interactive-user): connecting to 148.64.11.170
[08:06:15]: CAResp Tunnel#0(non-interactive-user) status:DENY-Request unrecognized
[08:06:16]: WSS connection denied by administrator, Internet access blocked
[08:06:16]: UDP Connection failed (ec:24 - The tunnel has been closed), will attempt TCP on the same DP
[08:06:17]: Couldn't establish connection due to error (ec:24 - The tunnel has been closed)

We confirmed from Firewall logs that the WSS Agent did send requests out to this destination, but handshake never completed else we would have seen a 'connected' message. 

When the WSS Agent authenticates itself to Cloud SWG, it needs to provide certain pieces of information including country code. If no country code exists, the GEOIP field is usually blank, but in this case the GEOIP field existed with country code and other fields being actually blank, as highlighted below. This is what caused the authentication to break.

"{"opMode":"ACTIVE","egressIP":"203.0.113.1","gatherPII":true,"blockIpv6Traffic":true,"unparsedProperties":"{\"allowQUIC\":false,\"allowUserDisable\":false,\"skipDisableReason\":false,\"cloudProxy\":\"199.19.250.205\",\"disableTamperProtection\":false,\"failOpen\":true,\"ignoreProxySettings\":true,\"promptForUpdate\":false,\"sendAllTraffic\":false,\"forwardPorts\":[80,443,8080,8443,7777,48010],\"uninstallToken\":[\"yyyyyyyyyyyyyyyyyy\",\"xxxxxxxxxxxxxxxxxxxxxxxxxxx\"]}","connectList":[{"name":"GCHZU","host":"cfs-g-gchzu.threatpulse.net","dc":"GCHZU","refDistanceMts":57763,"IP":"148.64.11.170"},{"name":"GITMI","host":"cfs-g-gitmi.threatpulse.net","dc":"GITMI","refDistanceMts":193740,"IP":"148.64.10.164"},{"name":"GDEFR","host":"cfs-g-gdefr.threatpulse.net","dc":"GDEFR","refDistanceMts":349496,"IP":"199.247.40.164"}],"sysConfig":{"clientConfig":[... Edited out ...."}]},"reason":"GEOIP","geo":{"countryCode":"","countryName":"","regionCode":"","regionName":"","city":"","postalCode":"","latitude":47.0,"longitude":8.0,"metroCode":"","accuracyRadius":100}}"

Checking the Maxmind database, we could confirm that these GEOIP fields were not populated for the egress IP address in question. Updating this information with Maxmind allowed the WSS Agent tunnel to come up. A Cloud SWG change was also pushed out to handle this edge case.