VCF Upgrade precheck fails with error "NSX planning phase precheck fails"

Products

VMware SDDC Manager / VCF Installer VMware Cloud Foundation VMware NSX

Issue/Introduction

VMware Cloud Foundation (VCF) upgrade precheck fails for NSX during the planning phase. Retrying the upgrade precheck results in a failure at the same stage with the following error: "NSX planning phase precheck fails"

SDDC Manager /var/log/vmware/vcf/lcm/lcm-debug.log reports a failure in resource validation:

DEBUG [vcf_lcm,0000000000000000,0000,actionExecutionId=87a6####,opId=9104####,taskId=f4d0####] [c.v.v.l.p.p.f.a.PrecheckAgentPollAction,AsyncTaskEngine142] Agent resource type: NSX, Precheck Result: PrecheckResult(taskId=aea6####, name=NSX Precheck Workflow, status=SUCCESSFUL, resourceResults=[ResourceResult(resource=Resource[resourceType=NSX_T_MANAGER, name=nsxw01.example.com], status=FAILED, validationResults=[ValidationResult(validationName=NSX planning phase precheck, validationStatus=FAILED, ERROR=[Notification(severity=ERROR, message=LocalizableMessage(id=com.vmware.vcf.error.runtime.upf.resource.validation.failed, defaultMessage=Failed to execute the resource validations))])])])

SDDC Manager /var/log/vmware/vcf/domainmanager/domainmanager.log reports credential issues:

ERROR [vcf_dm,0000000000000000,0000] [c.v.v.c.n.s.impl.NsxtWs1bServiceImpl,ForkJoinPool.commonPool-worker-#####] Failed to fetch the OIDC end point info for nsxw01.example.com {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}
com.vmware.evo.sddc.common.core.error.FederatedBrokerIdentityException: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}

SDDC Manager Platform Database reflects a failed inventory audit for the NSX cluster:

psql -h localhost -U postgres -d platform -c "\x" -c "select id,cluster_fqdn,configuration from nsxt where cluster_fqdn='nsxw01.example.com'"

Output:

-[ RECORD 1 ]-+-------------------------------------------------
id            | da3f####-####-####-####-########971d
cluster_fqdn  | nsxw01.example.com
configuration | ######################
			  |	"auditError": {
              |   "errorCode": "Failed to load NSX Cluster from the Inventory",
              |   "errorDetails": "error_message : Failed to load NSX Cluster from the Inventory, httpStatus : , error_code : 0"

Environment

VMware Cloud Foundation (VCF) 9.1
VMware NSX
VMware SDDC Manager

Cause

The issue is caused by a credential mismatch between the SDDC Manager database and the NSX Manager appliance.

The audit account is disconnected in SDDC Password Management, resulting in a 403 Forbidden error when fetching OIDC endpoint information.
The SDDC platform database reflects a stalled audit state ("Failed to load NSX Cluster from the Inventory"), which prevents the precheck from successfully validating NSX resources.

Resolution

To resolve this issue, remediate the audit credentials in NSX Manager and synchronize the SDDC Manager inventory.

Follow the below steps

Log in to the SDDC Manager UI and navigate to Developer Center.
Expand Credentials > GET /v1/credentials.
Set resourceType to NSXT_MANAGER and click Execute.
Review the stored audit password for the NSX Manager.
Update the audit password on the NSX Manager appliance to match the password stored in SDDC Manager. Refer to KB 314657
Verify the audit account is not expired. SSH to the NSX Manager VIP as admin and run:
```
get user audit password-expiration
```
Switch to root and check the account status:
```
st e
```
```
chage -l audit
```
Remediate the password in the SDDC Password Management UI.
Validate the nsxt table in the SDDC platform database. If the auditError persists, perform a rolling reboot of all NSX Manager VMs in the cluster to refresh the service state and clear the stalled inventory load error
Once the NSX Manager VMs are online and the cluster status is Stable, retry the VCF Upgrade Precheck.