NSX Password Policy Compliance Check Fails with 403 Forbidden Error due to Salt-Minion Credential Mismatch
search cancel

NSX Password Policy Compliance Check Fails with 403 Forbidden Error due to Salt-Minion Credential Mismatch

book

Article ID: 453905

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer VMware NSX

Issue/Introduction

Password Policy compliance checks in VMware Cloud Foundation (VCF) fail for NSX Manager nodes. When exporting the compliance result from the UI, the following error is returned:

{"error": true, "message": "403 Client Error: Forbidden for url: https://localhost/api/v1/node", "timestamp": #############}

The /var/log/syslog on the affected NSX Manager node also displays authentication failures:

INFO NSX #### [nsx@#### audit="true" comp="nsx-manager" level="INFO" logger="AuditingServiceImpl" subcomp="http" threadName="Processing request ########-####-####-####-############" username=salt-minion-###########@127.0.0.1] UserName=salt-minion-###########@127.0.0.1, ModuleName="ACCESS_CONTROL", Operation="LOGIN", Operation status="failure"
ERROR NSX #### [nsx@#### comp="nsx-manager" errorCode="MP403" level="ERROR" logger="NsxRestAuthenticationEntryPoint" subcomp="http"threadName="Processing request ########-####-####-####-############"] The credentials were incorrect or the account specified has been locked.
INFO salt-minion ####### -  [ERROR   ] hostname=localhost controller=password_policy product=vcf Request to https://localhost/api/v1/node failed with detailed error: Error Message: The credentials were incorrect or the account specified has been locked. | Error Code: 403 | Module: common-services
INFO salt-minion ####### -  [ERROR   ] hostname=localhost controller=password_policy product=vcf Failed to get node info: 403 Client Error: Forbidden for url: https://localhost/api/v1/node
INFO salt-minion ####### -  [ERROR   ] hostname=localhost controller=password_policy product=vcf Failed to determine if current node is NSX manager cluster leader. Error: HTTPError: 403 Client Error: Forbidden for url: https://localhost/api/v1/node. Full error details: HTTPError('403 Client Error: Forbidden for url: https://localhost/api/v1/node')
INFO salt-minion ####### -  Traceback (most recent call last):
INFO salt-minion ####### -    File "/opt/saltstack/salt/extras-3.10/config_modules_vmware/framework/utils/nsx_rest_utils.py", line 307, in is_nsx_manager_cluster_leader
INFO salt-minion ####### -      current_node_uuid = self._get_cached_node_uuid()
INFO salt-minion ####### -    File "/opt/saltstack/salt/extras-3.10/config_modules_vmware/framework/utils/nsx_rest_utils.py", line 194, in _get_cached_node_uuid
INFO salt-minion ####### -      node_info, node_errors = self.get_node_info()
INFO salt-minion ####### -    File "/opt/saltstack/salt/extras-3.10/config_modules_vmware/framework/utils/nsx_rest_utils.py", line 177, in get_node_info
INFO salt-minion ####### -      node_info = self._client.get_json(f"{NODE_API_BASE_URL}/api/v1/node")
INFO salt-minion ####### -    File "/opt/saltstack/salt/extras-3.10/config_modules_vmware/framework/clients/nsx/nsx_rest_client.py", line 263, in get_json
INFO salt-minion ####### -      self.raise_for_status(response, full_url)
INFO salt-minion ####### -    File "/opt/saltstack/salt/extras-3.10/config_modules_vmware/framework/clients/common/rest_client.py", line 375, in raise_for_status
INFO salt-minion ####### -      raise urllib3.exceptions.HTTPError(http_error_msg)
INFO salt-minion ####### -  urllib3.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://localhost/api/v1/node
INFO salt-minion ####### -  [ERROR   ] hostname=localhost Exception in control password_policy for check_compliance operation 403 Client Error: Forbidden for url: https://localhost/api/v1/node.
INFO salt-minion ####### -  [ERROR   ] {'compliance_config': {'vcf': {'password_policy': {'status': <ComplianceStatus.FAILED: 'FAILED'>, 'errors': ['403 Client Error: Forbidden for url: https://localhost/api/v1/node']}}}, 'metadata_grains': {'vcfops_resource_kind': 'nsxm', 'vcfops_resource_vcf_id': '########-####-####-####-############', 'vcfops_resource_id': '########-####-####-####-############', 'vcfops_resource_domain_id': '########-####-####-####-############'}}

Environment

VCF 9.1
NSX

Cause

The local salt-minion service, which handles configuration and password policy orchestration in VCF, has encountered a credential mismatch or configuration drift, preventing it from authenticating against the local NSX Manager API (/api/v1/node). This results in an MP403 Forbidden error during the compliance check.

Resolution

To resolve this, the salt-minion service must be stopped and disabled, allowing the automated configuration engine to re-provision the minion and recreate the necessary service account on the node.

  1. Log in to the affected NSX Manager node via SSH.
  2. Stop and disable the salt-minion service:
    sudo systemctl stop salt-minion.service
    sudo systemctl disable salt-minion.service
  3. Verify the service is stopped:
    sudo systemctl status salt-minion.service
  4. Wait approximately 15-20 minutes for the automated configuration engine to detect the change, re-provision the minion, and recreate the required service account.
  5. Re-run the Password Policy compliance check in the VCF Operations/SDDC Manager UI.
Powered by Wolken Software