Howto: Troubleshoot WSS Agent issues on MacOS

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Broadcom considers macOS a tier 1 operating system, meaning that we provide full feature and support SLA parity for both Windows and macOS. In addition, we have a very large, stable user population on macOS. With that in mind, changes by Apple to macOS over the last few years have introduced isolated pockets of instability for some customers. Broadcom has, and continues to work diligently with Apple to resolve all known issues in the macOS code base. When appropriate and feasible, we have worked around macOS defects unilaterally. This article summarizes some of those challenges and what to do in the unlikely event you encounter an issue with macOS.

Starting with macOS Big Sur, Apple changed how third party networking applications interface with the operating system by implementing a network extension architecture, removing the ability for applications to interface with the OS as kernel drivers. Many networking applications, including WSS Agent, required refactoring to switch to the new network extension architecture. The first version of WSS Agent to support the new macOS network extensions architecture was 7.2.1.

As a new system architecture for Apple, it introduced a number of issues within macOS that have impacted some customers using WSS Agent. To date, there have been 15 defects filed by Broadcom with Apple against network extensions. Over the same period of time, Broadcom has reported no defects against the equivalent Windows software, for comparison. As of the time of this writing, there are still issues acknowledged by Apple that remain open defects in macOS.

Examples of defects in macOS that impacted WSS Agent that were subsequently resolved by Apple:

• Errors where streams were closed by the operating system prematurely (Fixed in macOS 11.5)
• Errors with third-party enterprise applications when network extensions were loaded but inactive (Fixed in macOS 12.0)
• Errors where the OS sent malformed proxy requests (Fixed in macOS 12.0)
• Errors where VPN traffic was dropped when a network extension was loaded but inactive (Fixed in macOS 12.0.1)
• Errors loading network extensions on certain hardware (Fixed in macOS 12.0.1)
• Errors contacting proxy servers when VPN software is connected (scheduled to be fixed in macOS 12.2)
• Errors where network extensions disconnected after a long period of use (scheduled to be fixed in macOS 12.2)

Environment

WSS Agent on macOS.

Resolution

To ensure the quickest and most efficient resolution to your issue, please follow these steps:

1. Before opening a Broadcom or Apple support ticket, always try to reproduce your issue against the latest versions of macOS and WSS Agent. If the issue still presents itself, proceed to the next step. 
   • Be advised that Apple’s policy is to only evaluate issues present on the latest publicly-available version of macOS. Reported issues for anything but the latest public release will be rejected by Apple.
2. Use the wssa-diag.sh script to gather all the data needed for us to quickly diagnose the issue. Share the data with Broadcom by opening a support ticket.
   

If the data indicate a WSS Agent issue, your technical support agent will advise you on next steps, which might include a workaround while we work on a permanent fix. Rest assured that we will permanently address the problem as quickly as possible.

Possible Outcomes

Broadcom will analyze the data you provided and provide next steps. There are generally two possible outcomes:

The data indicate a WSS Agent issue:

Your Broadcom technical support agent will advise you on next steps, which might include a workaround while we implement a permanent fix. Rest assured that we will aggressively and permanently address the issue. 

The data indicate a macOS issue: 

If we determine the root cause to be an issue outside of WSS Agent, you will most likely be advised to open a support ticket with Apple. We will not ask you to perform this step unless we are reasonably certain that the issue is macOS related. If you do not have an Apple support contract we highly recommend you obtain one like you would for any other production technology.

The recommendation for you to open an Apple support ticket should not be interpreted as Broadcom placing sole care of the issue on you. We will continue to assist until a solution is available. Although Broadcom has a good relationship and reputation with Apple, it is clear from years of interactions that they place a higher priority on customer-submitted defects than to those submitted exclusively by vendors. To put it directly: Customer sourced issues are resolved quicker than vendor sourced issues. Broadcom is also happy to support you in direct conversations with Apple. 

Please follow these steps to submit a support ticket with Apple:

1. Gather debug (“bugreport”) information. To gather detailed information needed by Apple, perform the following tasks:

   • Download and extract the wssa-diag.sh run './wssa-diag.sh --bugreport' (enter the host admin password when prompted)

   • When prompted, download the following profiles (Apple ID will be required):

     • https://developer.apple.com/services-account/download?path=/iOS/iOS_Logs/NetworkDiagnostic.mobileconfig

     • https://developer.apple.com/services-account/download?path=/OS_X/OS_X_Logs/mDNSResponder.mobileconfig

   • Install each of the profiles by doing the following:

     • Double click on downloaded profile

     • Open System settings -> Privacy and security -> Profiles and double click to install the profile

   • Return to the wssa-diag.sh script and select reboot option.
     Note that you will only see when both the above profiles have been successfully installed

   • After reboot, run ./wssa-diag.sh again

   • After replicating the issue, click YES to start the data collection.
     Note that collection can take 15-30 minutes (progress bar provides more info on time needed).
     
     

2. Send this debug information to Broadcom technical support to verify that all the required data is present. Your support engineer will also be able to help you log your bug report with Apple (see the next step).
   
   
3. Contact Apple: It is important that you work directly with Apple to ensure the fastest possible resolution and because you have full control of, and access to, your environment.
   1. Log a bug report at https://bugreport.apple.com. This process requires an Apple ID.  The following information will be requested:
      • Operating System: Choose “macOS.”
      • Title: Be as descriptive as possible.
      • Problem area: Select “NetworkExtensionFramework” (be sure you do NOT select “Network Framework” - that is a separate component).
      • Type of issue: Generally should be “Incorrect/Unexpected Behavior”, but could be one of the other options if they are more accurate for your particular issue.
      • Describe the issue: Be as precise as possible, including step-by-step directions to reproduce. Your Broadcom support engineer can help you with this information.
      • Attach the system information report and sysdiagnose files. This is a VERY IMPORTANT step, as without them, Apple will reject the bug report. These files are collected as a part of step 3 above, and can be extracted by Broadcom technical support for you. Alternatively, you can follow the steps outlined on the https://bugreport.apple.com (click the question mark next to the files it is prompting you to upload) to gather these files yourself.
        
        
   2. After you log your bug report with Apple, it will be assigned an ID (in the format “FBXXXXXXX”). This ID is mapped within Apple to a “Radar” issue and is used to share your bug information across Apple teams as your issue is worked on. You can also share the FBXXXXXX with Broadcom to be logged and associated with your support ticket in our system.
      
      
4. Open an enterprise support ticket
   1. This is needed to give your issue “priority” - without a corresponding enterprise support case, your FB issue will likely go unnoticed within Apple.
   2. Make sure you include your FBXXXXXX ID when you open your enterprise support ticket so that Apple support engineers have direct access to your bug report and the attached diagnostic files.
      
      
5. Communicate updates from Apple to Broadcom
   • Learning of updates to your Apple support case keeps us informed and might provide us with useful information that can improve outcomes for other customers that may experience the same issue.
   • If Apple resolves the issue, please be sure to pass along details so we can update our records and close out your Broadcom support ticket.
     
     

Additional Information

As a best practice, both Broadcom and Apple recommend:

• Always run the latest version of macOS
• Always run the latest version of WSS Agent
• Have an Apple enterprise support contract in place before deploying Mac hardware into a production environment.
• Familiarize yourself with the Apple support process