Error 403 url_response on a URL that doesn't require credentials to be accessed

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

When testing a URL after add it the profile to the url_response probe, the next error is displayed: The server understood the request, but is refusing to fullfill it.

It needs to be noticed that this page is public accessible, and no authentication or credentials are required to access it.

Environment

Release : 20.3

Component : UIM - URL_RESPONSE

Cause

If the URL can be accessed from a public IP, and no authentication is required, the URL might be protected by cloudflare.

This can be easily confirmed using the browser developers tools.

Another option to confirm this, can be done using cURL command. For instance: curl <URL>

Replacing <URL> by the url with the issue.

That should return an HTML output, that if copied to a file and after set the .html extension to that file is loaded in a browser it will show something like next:

 

Resolution

If the URL is protected by cloudflare, it is highly likely that cloudflare is blocking the access to that URL to url_response.

If you are the owner of the site you should whitelist the IP address of the robot where the url_response is deployed.

This is discussed on next stackoverflow thread:

https://stackoverflow.com/questions/18500088/curl-load-a-site-with-cloudflare-protection