NSX-T manager data source credentials have been changed and now no metrics are collecting in Aria Operations for Networks

Products

• VMware Aria Operations for Networks

Issue/Introduction

After changing the credentials successfully for an NSX data source, metrics are no longer collecting after 2+ hours have passed.

 

When credentials are changed, a data source restart usually should not be required. We see in the logs 1 time submit of the credentials and 2 retries.

You may see the similar errors messages to the following in logs found on the platform and collector.

Platform:

Review the saasservice logs found in the /var/log/arkin/saasservice directory on the platform:

2025-02-18T18:25:30.338Z INFO jetty.server.Slf4jRequestLogWriter dw-790421 write:62 127.0.0.1 - - [18/Feb/2025:18:25:10 _0000] "POST /saastocollectordatalinkservlet HTTP/1.0"
 200 159 "-" "Java/THttpClient" 20217
2025-02-18T18:25:30.944Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-###### - POST /resttosaasservlet sendMessageToCollector:6188 wait over for collector msg
Type UPDATE_DATA_PROVIDER_CONFIGURATION requestId ######-###-###-###-######## for collector #####:######:saastocollectorservlet
2025-02-18T18:25:30.945Z ERROR vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-###### - POST /resttosaasservlet sendMessageToCollector:6209 Request timed out and no response received in timeout (40 Sec) window. Host could be unreachable or request is taking too long. customerId ##### collectorId ###### msgType UPDATE_DATA_PROVIDER_CONFIGURATION requestId ######-###-###-###-########
2025-02-18T18:25:30.945Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-###### - POST /resttosaasservlet removeFromIssuedRequest:5759 removing request ######-###-###-###-########
2025-02-18T18:25:30.945Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-###### - POST /resttosaasservlet removeFromIssuedRequest:5769 notification issued for collector request from removeFromIssuedRequest customerid ##### collectorid ###### msgType UPDATE_DATA_PROVIDER_CONFIGURATION requestId ######-###-###-###-########
2025-02-18T18:25:30.945Z INFO vnera.SaasListener.ServiceThriftListener_ServiceImpl dw-###### - POST /resttosaasservlet sendMessageToCollector:6220 sendMessage done. customerId ##### collectorId ###### msgType UPDATE_DATA_PROVIDER_CONFIGURATION requestId ######-###-###-###-######## errCode -8 errDetails 
2025-02-18T18:25:30.945Z ERROR SaasListener.config.GenericDPConnectionConfigProcessor dw-###### - POST /resttosaasservlet handleModify:899 handleModify failed on collector. collectorId: ######, customerId: #####, dpId: NSXT_######.###.###, opMode: UPDATE, result: SaasResult(errorCode:-8, errorDetails:, dataSource:NSXT, dpId:null)
2025-02-18T18:25:30.945Z INFO SaasListener.config.NSXTConnectionConfigProcessor dw-###### - POST /resttosaasservlet handleParentOperation:323 Result of UPDATE for customerId: #####, dpId: NSXT_######.###.### is SaasResult(errorCode:-8, errorDetails:, dataSource:NSXT, dpId:NSXT_######.###.###)
2025-02-18T18:25:30.945Z INFO jetty.server.Slf4jRequestLogWriter dw-###### write:62 ##.###.###.## - - [18/Feb/2025:18:24:50 _0000] "POST /resttosaasservlet HTTP/1.1" 200 165 "-" "Java/THttpClient/HC" 40105

Collector:

Review the collector logs found in the /var/log/arkin/collector directory on the collector in question:

2025-02-18T18:25:22.231Z INFO core.engine.SaasCommandProcessor collector-process-msg-exec-57 processMessage:716 sending response for requestId: ######-###-###-###-########, type:UPDATE_DATA_PROVIDER_CONFIGURATION, resultCode:0, timeTaken:31282
2025-02-18T18:25:22.232Z ERROR core.saascommunication.AbstractSaasListenerTransport Thread-6 receiveMessage:152 sending response threw exception. customerId: #####, collectorId: ######, msgType: UPDATE_DATA_PROVIDER_CONFIGURATION, reqId: ######-###-###-###-########
2025-02-18T18:25:22.232Z ERROR core.saascommunication.AbstractSaasListenerTransport Thread-6 receiveMessage:157 receive message processing threw exception. customerId: #####, collectorId: ######
2025-02-18T18:25:22.232Z WARN common.utils.CommonUtils Thread-6 logException:2578 receive message rpc call threw exception: 
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)

.........


2025-02-18T18:26:11.739Z INFO core.saascommunication.AbstractSaasListenerTransport Thread-6 receiveMessage:140 Received and processing message from SaasService for customerId: #####, collectorId: ######, msgType: UPDATE_DATA_PROVIDER_CONFIGURATION, reqId: ######-###-###-###-########
2025-02-18T18:26:11.739Z INFO core.saascommunication.AbstractSaasListenerTransport Thread-6 receiveMessage:143 Sending response for message from SaasService for customerId: #####, collectorId: ######, msgType: UPDATE_DATA_PROVIDER_CONFIGURATION, reqId: ######-###-###-###-########
2025-02-18T18:26:11.740Z INFO core.saascommunication.SaasListener Thread-6 receiveMessage:87 incoming message received: UPDATE_DATA_PROVIDER_CONFIGURATION requestId:######-###-###-###-######## tenantId:##### requestTimeout:40000, receiveMessage.type: UPDATE_DATA_PROVIDER_CONFIGURATION
2025-02-18T18:26:11.740Z INFO core.engine.SaasCommandProcessor collector-process-msg-exec-57 processMessage:251 SaasIncoming Message received. RequestId=######-###-###-###-########, Type=UPDATE_DATA_PROVIDER_CONFIGURATION, TenantId=#####
2025-02-18T18:26:11.741Z INFO core.dataprovidermanager.DataProviderManager collector-process-msg-exec-57 updateDataProviderConfiguration:879 Updating DP: NSXT_######.###.###, DataProviderConfigurationChangeType DEFAULT
2025-02-18T18:26:11.741Z INFO core.dataprovidermanager.DataProviderManager collector-process-msg-exec-57 stopDataProvider:821 Stopping DP NSXT_######.###.###, mode: false
2025-02-18T18:26:11.741Z INFO dataproviders.tasker.Tasker collector-process-msg-exec-57 shutdown:690 Shutting down

Environment

Aria Operations for Networks 6.10
Aria Operations for Networks 6.11
Aria Operations for Networks 6.12
Aria Operations for Networks 6.12.1
Aria Operations for Networks 6.13
Aria Operations for Networks 6.14

Resolution

In some situations, it may be necessary to restart the data source in order to reset the connection. If metrics are not collecting within 2 hours of re-entering the credentials:

In Aria Operations for Networks

1. Open Accounts and Data Sources
2. In the row for the NSX-T Manager data source for which metrics are not collecting:
   • Toggle Data Collection "OFF"
   • Toggle Data Collection "ON"