Collecting Debug Information Using Policy Diagnostic Probe

Products

• ISG Proxy
• ProxySG Software - SGOS

Issue/Introduction

Because of separate diagnostic settings and collection locations, PCAP, SSL debug and HTTP debug need to be start, stopped and gathered manually. Due to timing issues or capacity issues, acquisition of all data may be difficult.

From SGOS 7.2 onward, diagnostic probes make it possible to collect all of this information easily and simultaneously.

Resolution

Procedure overview:

• Use CPL to define the details of the logs to be gathered
• Create conditions for the above definition to work
• Add a PCAP stop policy if necessary

1. Use CPL to define the details of the logs to be gathered (Admin Guide - Define Probe)
   

   Defining which transactions to log and how
    
   define probe case_label
   Since multiple define probes can be created, give them unique names

   condition=condition_label
   This is the label to jump to if it matches the condition created in the policy

   target=system_log:log_level[,system_log:log_level]
   Subsystem range and log level for HTTP Debug log
   Decide the log level for each sub system
   
   transaction_type= proxy|admin|health-check|all
   Implemented from 7.3.15.1
   Generally, proxy is fine, but Health Can be specified if only check etc. is required.

   policy_trace={yes|no}
   Whether to take policy_trace

   limit=transaction_limit
   (Optional) Maximum number of transactions, default 30, max 50

   limit.reset=number_of_seconds
   After the limit is reached, it is reset at this number of seconds and counting starts from limit 0

   limit.session=session_limit
   (Optional) Maximum number of transactions in one session Counted at First IN Default 10, max = same as limit
    
   alert=alert_channel:{first|last|both}
   (Optional) Where to notify when the log is completed or when the limit is reached
   For alert_channel, syslog can be specified. Event-log cannot currently be specified
   You can specify whether to notify at the beginning or end, or both

   delivery={hold|syslog}
   (Optional) Where to send the log. By default, -hold
   Users refer to https://IP_address:port/Diagnostics/Traces/list-all or https://IP_address:port/Diagnostics/Traces.
   -syslog Send to syslog

   scope={session|transaction}
   (Optional) Default -transaction, Call probe Only log transactions that match the condition
   -session Call probe Record the entire session including transactions that match the condition Limited by session_limit

   expiry[.utc]=expiry_time
   How long this probe setting is valid

2. Create conditions for the above probe definition to work
   Define the condition to gather the data. For example, use the client IP, target URL, response code, exception ID, etc.
   
   
3. Add PCAP stop settings as necessary
   Stopping PCAP is a separate operation from define probe. Create a separate CPL layer and create a rule that STOPs if it matches condition 2.

Example: 503 is frequently returned on some sites. The issue occurs intermittently on any client passing through the Edge SWG.

In this example, we create a list of target URLs and log only if a 503 response is returned.
Policy Trace and PCAP are also gathered.

A maximum of 50 transactions are acquired, and the transaction stops once, but starts again after 1 second.
Continues to acquire until 20230130:2350 (UTC). (30th January 2023)

define condition my_traffic
condition=probeURL http.response.code=503
end

define condition probeURL
url.domain=example.com
url.domain=example.org
url.domain=example.net
end

define probe TestCase
condition=my_traffic
scope=transaction
target=http:all,ssl:all
policy_trace=yes
limit=50
limit.session=50
limit.reset=1
;alert=syslog:both
delivery=hold
expiry=20230130:2350
end

(Stopping a pcap) Create another CPL Layer
<diagnostic>
condition=my_traffic diagnostic.stop(pcap) ;If the condition matches my_traffic, it will stop the pcap

You need to start the packet capture before this error (event) occurs.
The packet capture will need to be configured as a rolling pcap, so that it does not stop due to maximum size for packet captures.

You can retrieve the diagnostic traces from the advanced URL https://<EdgeSWG_Address>:8082/Diagnostics/Traces

Browsing to Administration > Service Information > Packet Capture, you will see that Current state: Stopped. You may download it from this location by pressing the Download button.

Another example in which an error is returned to the client, but it is unclear whether it is returned from the web server or Edge SWG. It occurs on a specific client.

In this example, the target client IP is identified, and a log is generated if Edge SWG returns an Exception to the client.
All matching transactions are targeted.
Policy Trace and PCAP are also taken.
The contents of define probe are not changed.

define condition troubleCond
client.address=x.x.x.x condition=error50x
end

define condition error50x; The error returned from SG is defined as an exception, so use that. If it matches any of the following, the condition becomes true.
exception.id=tcp_error
exception.id=ssl_failed
exception.id=ssl_client_cert_expired
exception.id=ssl_client_cert_ocsp_check_failed
exception.id=ssl_client_cert_ocsp_status_unknown
exception.id=ssl_client_cert_untrusted_issuer
exception.id=ssl_client_cert_revoked
exception.id=configuration_error
exception.id=internal_error
exception.id=authentication_failed
exception.id=content_filter_denied
exception.id=gateway_error
exception.id=invalid_request
exception.id=invalid_response
exception.id=dns_unresolved_hostname
http.response.code=503 ; This is for the response code value returned by the web server
end

define probe TestCase
condition=troubleCond
scope=session
target=http:all,ssl:all
policy_trace=yes
limit=50
limit.session=50
limit.reset=1
delivery=hold
expiry=20230130:2350
end

Create another CPL Layer
<diagnostic>
condition=troubleCond diagnostic.stop(pcap) ;If troubleCond is matched, the PCAP will be stopped.

When the conditions are matched a log will be generated in the advanced URL /Diagnostics/Traces location.