How to download URLs/Domains in URL list object

Products

• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Cloud SWG managed via Cloud SWG Portal and / or Enterprise Console.

Cloud SWG admin using policy user defined object lists with up to 1000s of entries in each list.

When looking at list entries via the Cloud SWG / Enterprise Portal, there are no options to export or view more than one page of entries at a time.

How can I view all the list entries easily, or batch update entries in the list in a simple manner?

Environment

Cloud SWG.

Enterprise Console.

Policy object lists.

Cause

Cannot be done from the Portal.

Resolution

Need to use Policy list API instead of Cloud SWG to get the info needed.

By querying the lists endpoint as shown below, one can get all the available policy lists on the tenant (replace ###### with base64 encoded username:password string):

$ curl --location 'https://portal.threatpulse.com/api/rest/policy/lists' --header 'Authorization: Basic ######'

[{"name":"IP Bypass List","id":"12345","type":"destination_ip"},{"name":"ExampleURLs","id":"23456","type":"url"},....,{"name":"Manual banned list","id":"34567","type":"geolocation"}]

If we want the entries in a specific list e.g. all URLs in the ExampleURLs list, run the same query but reference the list ID as shown here:

$ curl --location 'https://portal.threatpulse.com/api/rest/policy/lists/23456' --header 'Authorization: Basic ######' 

 

Technically, when logged into the Portal and browsing the API object lists, a HAR file will capture the requests that trigger the list entries displayed on the Cloud SWG Portal. Looking closely at these requests, such as the one below, they are always POSTs to https://portal.threatpulse.com/djn/directprovider endpoint and pass Cloud SWG Portal session information as well as two parameters that can be used to automate the process (startindex and pagesize). By right clicking the request in the HAR file and 'copy as cURL', the request attributes look as follows:

curl 'https://portal.threatpulse.com/djn/directprovider' \
  -H 'accept: */*' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -b 'BCSI-ACPS-0000000000000000=#####; JSESSIONID=#####; TPLoadType=null' \
  -H 'origin: https://portal.threatpulse.com' \
  -H 'pragma: no-cache' \
  -H 'priority: u=1, i' \
  -H 'referer: https://portal.threatpulse.com/' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'x-requested-with: XMLHttpRequest' \
  --data-raw '{"action":"PolicyServices","method":"getListWithUsages","data":["URL_LIKE",{"startIndex":0,"pageSize":20,"sortingOrder":[{"property":"name","ascending":true}]},true],"type":"rpc","tid":89}'

By crafting a script that runs the request multiple times, incrementing the startIndex by 20 each time, every list entry can be downloaded to a file.