Broadcom started migrating all our Web Security Service (WSS) Cloud Reporting Service customers from AWS to GCP. After the migration, a small subset of customers using the WSS SyncAPI may find an edge scenario where some residual data will still be trickling to AWS, while the bulk of the data will flow to GCP. In this case, the WSS SyncAPI will serve the data still in AWS until it is completely drained, and not serve the data already in GCP. No data is lost in this scenario.
Web Security Service
WSS Cloud Reporting Service
WSS SyncAPI customers encountering this problem should reset the token used in the SyncAPI to "none" ("token=none"):
curl -k -vvv "https://portal.threatpulse.com/reportpod/logs/sync?startDate=1620867600000&endDate=0&token=none" -H "X-APIUsername:<api user name>" -H "X-APIPassword:<api password>" -o WSS_log.zip
With the next SyncAPI request, a new token will be served and customers will start receiving their full data set as expected. Data from AWS will continue to migrate to GCP and will be served with normal WSS SyncAPI requests, albeit with a delay, until the WSS service completely stops sending access log data to the old system in AWS.
Note: Date parameters are GMT in milliseconds-since-1970
Timestamp in milliseconds: 1620867600000
Date and Time (GMT): Thursday, May 13, 2021 1:00:00 AM
Customers can download the data that was missed by the WSS SyncAPI due to this issue by using the WSS Download API together with the sample bash script provided below for download, as described below:
Example script execution:
wss-download-backfill.sh <api-user-name> <api-password> hour "2021-05-13 01:00" "2021-05-13 23:00"
Output: A compressed archive containing the logs for the dates provided in the script paramters.
Attached to the article is a sample script. Customers are free to adapt the sample script to their needs. Once downloaded, make the script executable using the command:
chmod +x 1620938959189__wss-download-backfill.sh