Flows are not being displayed on Aria Operation for Networks for an NSX manager connection, while using NSX custom local accounts

Products

• VMware NSX

Issue/Introduction

• In Aria Operations for Networks (AON) a connection to NSX is setup, using a customer local account and the NSX manager cluster VIP.
• On the AON collector log: /var/log/arkadin/collector/collector.STDOUT-####-##-##-##.log the follow error is seen:

ERROR dataprovider.utils.HttpUtils Task_NSXT_nsx-manager-vip.example.com-0 checkStatusAndThrow:41 API /api/session/create error response {"module_name":"common-service","error_message":"Authentication Failed: Invalid credentials","error_code":98}
WARN common.utils.CommonUtils Task_NSXT_nsx-manager-vip.example.com-0 logException:2678 Exception occurred while getting access token for dpId = NSXT_nsx-manager-vip.example.com.
com.vnera.dataproviders.core.common.impl.dataprovider.utils.exceptions.HttpException: Could not get response for /api/session/create, status 403

• On the NSX manager which is the VIP leader, in the log /var/log/proxy/envoy_access_log.txt we see the following API calls from the collector:

<vRNI_Collector_IP_Address> <NSX_Manager_VIP_Address> "POST" "/api/session/create" "HTTP/1.1" 403 UAEX 60 109 220 - "<vRNI_Collector_IP_Address>" "Apache-HttpClient/4.5.9 (Java/17.0.10)" "05b05c9c-####-####-####-88931025b938" "<NSX_Manager_VIP_FQDN>:443" "-"
<vRNI_Collector_IP_Address> <NSX_Manager_VIP_Address> "GET" "/api/v1/global-configs/OperationCollectorGlobalConfig" "HTTP/1.1" 403 UAEX 0 141 197 - "<vRNI_Collector_IP_Address>" "Apache-HttpClient/4.5.9 (Java/17.0.10)" "acd37d72-####-####-####-c644d94abc83" "<NSX_Manager_VIP_FQDN>:443" "-"

• On the NSX manager which is the VIP leader, in the log /var/log/proxy/reverse-proxy.log we see the following:

INFO grpc-default-executor-11390 HttpClientUtil 2970 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Making request to http://127.0.0.1:6565/api/session/create
INFO Processing request 05b05c9c-###-###-###-88931025b938 PAMAuthenticationProvider 2970 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Local auth for <custom_local_user_account> unsuccessful, trying other auth methods.

• Opening the UI to each NSX manager and logged in as admin user, looking at System, User Management, Local Users, all 3 managers have a different view and the manager which is the VIP leader, does not list the <custom_local_user_account> shown in the reverse_proxy.log above.
• Running start search resync all on all 3 NSX managers, does not resolve the issue.

Environment

VMware NSX 4.1.x

Aria Operations for Networks

Cause

The NSX manager holding the VIP, does not have a correct view of all the local users. Normally the 3 managers are clustered and each manager has the same information as the other. Due to the issue in KB NSX credentials are not being synchronized between NSX Managers after manual password reset, the information in each manager node can be inconsistent.

Resolution

Apply the workaround mentioned in the KB NSX credentials are not being synchronized between NSX Managers after manual password reset.

To workaround the issue, you can also move the VIP leader to the NSX manager which does list the custom_local_user_account which is the API call is using and failing.

Note: To move the VIP, you need to stop the proxy service on the manager, this will then cause the VIP to move to another manager, if this is not the desired manager, you may need to stop the proxy on the second manager also, once the VIP is on the desired manager, you can start the proxy service again on the others.

To stop the proxy service, log in as root user on the NSX manager and run:

/etc/init.d/proxy stop

To start the proxy service, log in as root user on the NSX manager and run:

/etc/init.d/proxy start

Additional Information

NSX credentials are not being synchronized between NSX Managers after manual password reset