VCF 9.1.1.0 Upgrade Fails with Resource Validations Timed Out and 403 Unauthorized

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• An upgrade pre-check from VMware Cloud Foundation (VCF) 9.1.0.0 to VCF 9.1.1.0 fails during the VMware NSX planning phase.
• The SDDC Manager UI explicitly reports the following error under the NSX planning phase precheck: Resource validations timed out.
• During this time, the local NSX admin account is also actively disconnected and unable to authenticate to the NSX UI or API.
• Subsequent attempts to select "Apply Configuration Upgrade" fail entirely.
• The UI returns an explicit 403 Unauthorized error detailing a lockout or credential mismatch:
• Error Message: Failed to add SDDC Manager IP to NSX lockout immune addresses for NSX cluster <######>. Error: Unauthorized (com.vmware.vapi.std.errors.unauthorized) (statusCode:403) => (messages = [], data = => (error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services), errorType = UNAUTHORIZED, challenge = <null>)Remediation message: Verify that the NSX Manager is accessible and the credentials are valid, then try again.
• Further investigation into the NSX audit logs reveals a continuous loop of authentication failures originating from the SDDC Manager <######>.
• The audit logs on NSX display repetitive warning messages similar to the following snippets, indicating thousands of rapid, consecutive failed login attempts:
• /var/log/nsx-audit.log
WARNING NSX logger="AuthenticationBlacklistService" subcomp="http" threadName="Processing request..." Account svc-nsx-sddc... has been temporarily locked for 900 seconds after 1792 consecutive failed login attempts.

Environment

VMware Cloud Foundation (VCF) 9.1.0.0

VMware Cloud Foundation (VCF) 9.1.1.0

VMware NSX-T

Cause

SDDC Manager continuously attempts to authenticate to the NSX Manager using stale or out-of-sync cached credentials, creating an authentication loop that triggers the 900-second NSX lockout mechanism on the service accounts and concurrently disconnects the NSX admin account. Even after successfully rotating the credentials via the SDDC Manager API, the Operations Manager service continues utilizing the stale cached payload.

Resolution

1. Implement the lockout immunity workaround steps documented in KB 449684.

2. Stop the NSX API Service on the primary NSX-T Manager node to prevent peer-node overwrites (Verify success by ensuring the status returns a stopped state):/etc/init.d/nsx-mp-api-server stop

3. Force a Cluster Credential Sync by resetting the admin password and creating the synchronization trigger flag:passwd admintouch /var/vmware/nsx/reset_cluster_credentials

4. Restart the NSX API Service to initiate cluster-wide synchronization:/etc/init.d/nsx-mp-api-server start(Note: If this fails to propagate, log into NSX nodes 2 and 3 and manually run passwd admin to match node 1).

5. Clear the stale SDDC Manager cache by logging into SDDC Manager via SSH (as root) and restarting the Operations Manager service:systemctl restart operationsmanager

Additional Information

NSX service account disconnected and password rotation failure in SDDC Manager