Auth Connector fails to connect to connect to data pods that users for that tenant have not accessed before

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Users accessing the internet via Cloud SWG using the WSS Agent access method.

On a few occasions (after WSS Portal maintenance), users complain about getting blocked from 4accessing sites they should normally be able to access.

Each block is related to the user missing their group information.

Looking at the state of the Auth Connector connections at the time, we always see 0 / 0 active connections.

netstat output from Auth Connector at the time clearly shows many ESTABLISHED connections into Cloud SWG pods.

Manually restarting the Auth Connectors would always fix the issue.

Environment

Auth Connector

Cause

In certain upgrade/reboot flows, Auth Connectors were connecting to the backup Portal B (while Portal A was briefly unreachable) and getting stuck there (i.e. connected but not usable), resulting in failures to notify the Auth Connectors of newly accessed data pods for that tenant.

 

Resolution

Portal code change (July '22) needed to fix issue. Auth Connector now getting notified of new pods when Portal restarted. 

Additional Information

Could really only troubleshoot from the back end using internal tools.

ACs should get notified of any new pods that tenant users are sent to via the comms channel into Portal. As soon as notification happens, the AC will trigger a TCP connection request into the new pods and exchange the info needed to get user and group information. Since the AC was never notified, the AC debug logs never showed up anything.