url_response probe monitoring fails with 403 error.

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

We are trying to monitoring few external websites, Probe is failing with Error: 403. We are able to connect to same websites from server without any issues. Please find the attachment for error message.

Environment

Release : 20.3

Component : UIM - URL_RESPONSE

Resolution

First,

In the url_response probe profile Properties, go to the "Authentication" tab, and ensure that the "Windows NT authentication" is disabled.

Test again.

Second, if it still fails, then:

Verify the type of security that is required to connect to the website. The url_response probe does not support ADFS authentication (Active Directory Federation Services). The probe does support Windows NT authentication along with the following authentication types: 
 
BASIC
NTLM
DIGEST
GSSNEGOTIATE
ANYSAFE

Lastly, if the URL is protected by cloudflare, it is highly likely that cloudflare is blocking the access to that URL to url_response.

If you are the owner of the site you should whitelist the IP address of the robot where the url_response is deployed.

This is discussed at this stackoverflow url thread:

https://stackoverflow.com/questions/18500088/curl-load-a-site-with-cloudflare-protection

Try adding the machine's ip address where the url_response probe is deployed, to the cloudflare 'white list.'

See also: https://articles.assembla.com/en/articles/1623119-certificate-verification-error-20-unable-to-get-local-user-certificate

If the issue persists consider that this is a server issue when trying to contact it via use of curl which is what url_response uses under the covers.

It could be that curl is blocked as a user-agent string in Firewall Rules so you will have to check with the server/firewall/security team. There is nothing we can do on the url_response side as its designed to use curl.

curl -skvo /dev/null https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct 6 --connect-to ::###.##.##.##
* Connecting to hostname: ###.##.##.##
*   Trying ###.##.##.##:443...
* Connected to ###.##.##.## (###.##.##.##) port 443 (#0)
* schannel: disabled automatic use of client certificate
* schannel: ALPN, offering http/1.1
* schannel: ALPN, server accepted to use http/1.1
> GET /cdn-cgi/beacon/expect-ct HTTP/1.1
> Host: report-uri.cloudflare.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Date: Thu, 09 Jun 2022 13:16:29 GMT
< Connection: keep-alive
< Server: cloudflare
< CF-RAY: 718a24597fdc1a0b-EWR
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
<
* Connection #0 to host ###.##.##.## left intact
* Connecting to hostname: ###.##.##.##
*   Trying ###.##.##.##:80...
* Connected to ###.##.##.## (###.##.##.##) port 80 (#1)
> GET / HTTP/1.1
> Host: 0.0.0.6
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Date: Thu, 09 Jun 2022 13:16:29 GMT
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 16
< Connection: close
< X-Frame-Options: SAMEORIGIN
< Referrer-Policy: same-origin
< Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Expires: Thu, 01 Jan 1970 00:00:01 GMT
< Server: cloudflare
< CF-RAY: 718a245a39de8c12-EWR
<
error code: 1003* Closing connection 1