VCF Operations for Networks Data Collection from NSX Manager fails due to Service Account Password Changes

Products

• VCF Operations for Networks
• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• Impacts when the password for the VCF Operations for Network service account, used to connect to the NSX-T Manager as a data source, is changed.
•  "No Data being Received" status 
• Collector unresponsiveness to ICMP pings.

• Following a password change  for the data source user will be unable to retrieve data from NSX.
• User will observe authentication failure requests in the NSX Manager reverse-proxy.log, and the envoy_access_log.txt will display a 403 response code, indicating an authentication issue on the client side.
• Eventually the UI will present an error message stating "Invalid credentials"

• Review the NSX Manager /var/log/proxy/reverse-proxy.log will log a message similar to the following:

Account <service account name> has been temporarily locked for 900 seconds after 5 consecutive failed login attempts.

• In the Collector log located at /var/log/arkin/collector You may see the following ERROR log events:

YYYY-MM-DDTHH:MM:SS.SSSZ ERROR dataprovider.utils.HttpUtils Task_NSXT_<nsx manager fqdn>-0 checkCodeAndThrow:53 Could not get response for /policy/api/v1/infra/ipfix-dfw-collector-profiles/vrni-ipfix-collector-profile, status 404
YYYY-MM-DDTHH:MM:SS.SSSZ ERROR dataprovider.utils.HttpUtils Task_NSXT_<nsx manager fqdn>-0 checkStatusAndThrow:41 API /policy/api/v1/infra/ipfix-dfw-collector-profiles/vrni-ipfix-collector-profile error response {
YYYY-MM-DDTHH:MM:SS.SSSZ ERROR vmware.nsxcommon.AbstractClusterDataSourceFetchUtils Task_NSXT_<nsx manager fqdn>-0 executeHttpCall:136 Error calling callable for path /policy/api/v1/infra/ipfix-dfw-collector-profiles/vrni-ipfix-collector-profile
YYYY-MM-DDTHH:MM:SS.SSSZ ERROR tasks.datafetchers.PolicyIpfixStateWatcher NSXT_<nsx manager fqdn>_Config_OpMgr_Policy-0 sendMultipleIPFIXCollectorEventSDM:118 vRNI CollectorIPs List is null or empty, vrniCollectorIPs: [x.x.x.x]

YYYY-MM-DDTHH:MM:SS.SSSZ ERROR dataprovider.utils.HttpUtils NSXT_<nsx manager fqdn>_Config_OpMgr-1 checkCodeAndThrow:53 Could not get response for /api/session/create, status 403
YYYY-MM-DDTHH:MM:SS.SSSZ ERROR dataprovider.utils.HttpUtils NSXT_<nsx manager fqdn>_Config_OpMgr-1 checkStatusAndThrow:41 API /api/session/create error response {"module_name":"common-service","error_message":"Authentication Failed: Bad credentials","error_code":98}
YYYY-MM-DDTHH:MM:SS.SSSZ ERROR dataprovider.utils.HttpUtils NSXT_<nsx manager fqdn>_Config_OpMgr-6 checkCodeAndThrow:53 Could not get response for /api/session/create, status 403
YYYY-MM-DDTHH:MM:SS.SSSZ ERROR dataprovider.utils.HttpUtils NSXT_<nsx manager fqdn>_Config_OpMgr-6 checkStatusAndThrow:41 API /api/session/create error response {"module_name":"common-service","error_message":"Authentication Failed: Account is temporarily locked","error_code":98}

 

Environment

• VCF Operations for Networks 6.13
• VCF Operations for Networks 6.14.x
• VMware NSX 4.x

 

Cause

The password for the Service account that was used to authenticate was changed / updated which resulted in this issue.

Resolution

To resolve this issue follow the steps below:

1. To restore data collection, you must update the password for the service account used for NSX in the VCF Operations for Networks UI.
2. In some cases it is necessary to power cycle the collector node via vCenter if the node is unresponsive.