Identify And Resolve An Expired Internal Certificate

Products

• Web Isolation

Issue/Introduction

The system's components in web isolation gateways use an internal trust mechanism, based on internal certificates and self-signed CAs.  These are uniquely generated per installation, during the first-time wizard execution.

 

If these certificates have expired, push settings will fail and the gateways will show "Error" status  and "Timed out" in the details column.

 

Environment

This applies to web isolation on-premises only.

Cause

In a very early product version, these certificates were set to expire 5 years after the first-time-wizard run.

Resolution

How To Check For Expired Internal Certificate

For GW / Management:

• SSH to the problematic gateway
• Search for the “async_services_storage” folder as follows
  • find . -name async_services_storage
• openssl x509 -enddate -noout -in  <path_to_async_services_storage>/ca_certificate.pem
• See if the output date is older than today. If that is the case, then the issue is in fact an expiration of an internal certificate

 

For PDP:

• SSH to the problematic PDP
• Search for the “pdp_storage” folder as follows
  • find . -name pdp_storage
• openssl x509 -enddate -noout -in <path_to_pdp_storage>/ca_certificate.pem
• See if the output date is older than today. If that is the case, then the issue is in fact an expiration of an internal certificate

 

Temporary Workaround Patch

Download the patch via any Web Isolation GW (even in a test tenant) with the following fgcli command:

• fgcli fileserver download patch/Fixes/Internal-certificate/pdp_internal_certificate_patch.sh /tmp/pdp_internal_certificate_patch.sh

 

Run the patch on every gateway

• Make sure that you have the password for sudo operation on the gateway
• Enter in the following syntax command:
  • chmod +x /tmp/pdp_internal_certificate_patch.sh && /tmp/pdp_internal_certificate_patch.sh

 

Revert Temporary Workaround Patch (only if needed!)

Download the patch removal with the following fgcli command:

• fgcli fileserver download patch/Fixes/Internal-certificate/pdp_internal_certificate_de_patch.sh

 

Run the patch removal on every gateway

• Change mode and run the patch removal on all the gateways in which you want to revert the patch
  • chmod +x /tmp/pdp_internal_certificate_de_patch.sh && /tmp/pdp_internal_certificate_de_patch.sh

 

***Note: this is a temporary workaround that will not survive a version upgrade***

 

Permanent Solution

The workaround described above is only used as an immediate mitigation to stabilize the environment.  The proper and permanent solution is to run the manual certificate renewal process.  However, this should only be done by the Web Isolation Customer Focus team and not by Broadcom Support.

Please open a ticket with Broadcom Support and a session will be scheduled with Web Isolation Customer Focus team to run the permanent fix.

Additional Information

It's important to stress this is not a security vulnerability, but rather an operational issue, which results in disruption of the Web Isolation service.

This is not related to the man-in-the-middle CA certificate used in Web Isolation, but rather to the internal communication self-signed CA public key.

This issue is only relevant for on-premises customers. For cloud deployments we have ensured that no customer is impacted.