Detection Server WebserverScanner url rejected due to robot protocol

Products

• Data Loss Prevention Discover Suite

Issue/Introduction

When scanner a webserver with the SymantecWebServerScanner in 15.8 a failure occurs. 

in the C:\Program Files\Symantec\DataLossPrevention\DetectionServer\15.8.00000\Protect\Scanners\WebServerScanner\scanner\logs\synchronize.log 

16/07/2021 12:09:57 [29] 10-Full: 101_122: Url rejected due to robot protocol: https://192.168.1.160:8444/index.php/

 

Environment

Release : 15.8

Component : SymantecWebServerScanner

Cause

the webserver is not allowing bots to crawl the website 

Resolution

on the webserver that is being scanned; the robots.txt needs to allow the detection server to be able to scan. 

change the robots.txt from 

User-agent: *
Disallow: /

to 

User-agent: *
Disallow: