Web Console Unavailable Due to IIS 500/503 Errors

Products

• Carbon Black App Control (formerly Cb Protection)

Issue/Introduction

• Issues start after triggering large amount of agent upgrades, refreshing policies on multiple agents at once, installing new rules package, or after recent server certificate update
• Web console pages fail to load with errors:

  500 Internal server error
  503 Service is unavailable

• IIS logs shows lost of GET requests with 500/503 status:

  2022-11-15 03:43:12 10.11.12.13 GET /hostpkg/pkg.php pkg=configlistwithdeleteditems.xml.egk 443 - 10.22.33.44 Bit9+Parity - 500 0 64 81986
  2022-11-15 03:43:12 10.11.12.13 GET /hostpkg/pkg.php pkg=ParityHostAgent.msi 443 - 10.22.33.44 Bit9+Parity - 500 0 64 41986
  2022-11-15 03:43:12 10.11.12.13 GET /hostpkg/pkg.php pkg=TrustedCertList.pem 443 - 10.22.33.44 Bit9+Parity - 503 4 64 41986
  2022-11-15 03:43:12 10.11.12.13 GET /hostpkg/pkg.php pkg=Yara.bt9 443 - 10.22.33.44 Bit9+Parity - 503 4 64 41986 

• Windows event logs may show:

  Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures.

Environment

• App Control Server: All Supported Versions
• Microsoft Windows Server: All Supported Versions
• Microsoft IIS: All Supported Versions

Cause

• The Default App Pool is crashing or is unavailable

Resolution

1. Upgrade the App Control Server to version 8.10.4 or higher and make the following changes:
   1. Navigate to System Configuration > Advanced Options
   2. Update the Resource Download Location to match the Certificate Download Location, for example:

      Resource Download Location: https://**ServerIP**/packages/
      Certificate Download Location: https://**ServerIP**/packages/

   3. Open IIS Manager > Sites > Parity Console Web > MIME Types Icon:
   4. Add the following details:

      File name extension: .egk | MIME type: application/x-egk-file
      File name extension: .enc | MIME type: application/x-enc-file
      File name extension: .bt9 | MIME type: application/x-bt9-file

2. If the issue still persist, please collect server logs

Additional Information

If unable to upgrade to 8.10.4+ at this time, make the following changes:

1. Open IIS manager (inetmgr) > Expand the web server name in the left menu options
2. From the left menu, right-click on "Application Pools" > right-click the "DefaultAppPool" > Advanced Setting
3. In the Advanced Settings, scroll down and update the following:
   • Process Model > Identity: Use the App Control Service Account
   • Rapid-Fail Protection > Maximum Failures: 10
   • Recycling > PrivateMemoryLimit: 0
4. Select the web server name in the left menu options
5. Select "FastCGI" Settings
6. Edit "C:\Program Files (x86)\Bit9\Parity Console\php\php-cgi.exe"
7. Verify and update the following:

• Instance MaxRequests: 10000
• Max Instances: 0
• Queue Length: 10000

8. Restart the IIS Web Server from CMD: iisreset