In vRealize Automation Portal NSX Endpoint validation fails with "Unable to connect to endpoint. The credentials are invalid." error.

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Validating an NSX endpoint fails intermittently
• You verified credentials are correct
• You see an error:

"Unable to connect to endpoint. The credentials are invalid."

• NSX related catalog items fail to provision in vRA
• In /var/log/vmware/vcac/catalina.out log you see entries similar to:

[UTC:2018-09-06 17:25:12,177 Local:2018-09-06 17:25:12,177] vcac: [component="cafe:iaas-proxy" priority="INFO" thread="tomcat-http--67" tenant="tenant-name" context="7bn20qHL" parent="7bn20qHL" token="D1J3zJ3X"] com.vmware.vcac.iaas.vco.network.impl.NsxRestTemplateFactory.lambda$getRestTemplate$0:129 - Making [GET] NSX API Rest call to URL [https://NSX-manager/api/2.0/services/usermgmt/scopingobjects] with body []
[UTC:2018-09-06 17:25:12,252 Local:2018-09-06 17:25:12,252] vcac: [component="cafe:iaas-proxy" priority="INFO" thread="tomcat-http--67" tenant="tenant-name" context="7bn20qHL" parent="7bn20qHL" token="D1J3zJ3X"] com.vmware.vcac.iaas.vco.network.impl.NsxRestTemplateFactory.handleError:248 - Handling error. status code: [403], raw status code [403], details [Forbidden]
[UTC:2018-09-06 17:25:12,254 Local:2018-09-06 17:25:12,254] vcac: [component="cafe:iaas-proxy" priority="INFO" thread="tomcat-http--67" tenant="tenant-name" context="7bn20qHL" parent="7bn20qHL" token="D1J3zJ3X"] com.vmware.vcac.iaas.controller.endpointconfiguration.EndpointController.validate:196 - Endpoint with name: [NSX-endpoint] validation complete with result [INVALID_CREDENTIALS] .
 

• Attempt to access https://NSX-manager/api/2.0/services/usermgmt/scopingobjects API ​results in failure

Status 403 – Forbidden 
This IP address has been blocked temporarily. 
The server understood the request but refuses to authorize it. 
 

Environment

VMware vRealize Automation 7.x

Cause

A common cause of this issue is a monitoring tool that is frequently attempting to login to run 'GET' API calls with the incorrect credentials

Resolution

Identify the source of the REST API call and eliminate it.

This device can be tracked down in the local_access.log of the NSX manager in the /usr/appmgmt-tcserver/logs/ location. Within this log, you will see numerous attempts with a 403 response as shown below:

10.10.xx.xx - - [11/Sep/2018:07:20:53 +0000] "GET /api/2.0/services/securitygroup/securitygroup-116 HTTP/1.1" 403 1014 2556

10.10.xx.xx - - [11/Sep/2018:07:20:54 +0000] "GET /api/2.0/services/usermgmt/scopingobjects HTTP/1.1" 200 26936 227