Certificate tasks fail in VMware Cloud Foundation 9.x with "Received invalid response Field level validation errors"

Products

VMware NSX VMware Cloud Foundation VMware SDDC Manager / VCF Installer

Issue/Introduction

SDDC Manager certificate replacement tasks may fail during CSR generation or validation phases or during a certificate import task.

Example of Certificate Operation: Generate CSR subtask failure:

Description

Generate CSR for ####FQDN#### and resource type nsx

Progress Messages Failed to generate CSR for ####FQDN#### due to: Received invalid response Field level validation errors: {value '####FQDN####' of property extensions.subject_alt_names.dns_names[0] violates format 'san-entry-dns' for ####FQDN####

Error

Message: Failed to generate CSR for ####FQDN#### due to: Received invalid response Field level validation errors ####shortname####' of property extensions.subject_alt_names.dns_names [0] violates format 'san-entry-dns' for ####FQDN####

Remediation Message:

Reference Token:

Cause:

Import in SDDC Manager of certificates with CSRs generated by NSX Manager and signed by VMCA or another CA also fail.
CSRs generated by SDDC Manager for NSX nodes incorrectly include shortnames (example: "nsx_mgr1" vs. FQDN of "nsx_mgr1.mydomain.org") in the Subject Alternative Name (SAN) fields, even if the environment uses FQDNs.
VCF 9 security guardrails reject the resulting certificates, citing identity mismatches.
Attempting to change the hostname via the NSX admin CLI (KB 386603 - How to change NSX Manager Appliance FQDN) does not update the CorfuDB record and will not resolve the lifecycle failure.
If set hostname <FQDN> has been used on the NSX Manager admin CLI, then the command get hostname should return the same FQDN. However, the return of API GET /api/v1/cluster/nodes/deployments shows shortnames in the "hostname" field.
- This can be checked on the NSX Manager root CLI directly with command:
```
curl -k -u admin -H "Accept: application/json" -H "Content-Type: application/json" https://127.0.0.1/api/v1/cluster/nodes/deployments
```

Environment

VMware Cloud Foundation 9.x
NSX 9.x
VMware SDDC Manager 9.x

Cause

NSX Manager nodes that were originally deployed using shortnames (common in legacy or brownfield NSX-T installs) store that shortname as an immutable identity in the CorfuDB deployment record (/api/v1/cluster/nodes/deployments).
When SDDC Manager in VCF 9.x performs certificate rotation, it queries the NSX API to build the CSR spec.
Because SDDC retrieves the NSX shortname from the deployment metadata, the rotation then fails VCF 9's strict FQDN validation guardrails.

Resolution

To resolve the identity mismatch and allow SDDC Manager to successfully rotate certificates, the NSX Manager nodes must be redeployed to update the underlying database metadata.
Perform a rolling replacement of the NSX Manager cluster nodes, ensuring each node is redeployed using its Fully Qualified Domain Name (FQDN) in the deployment parameters. This can be done prior to upgrading NSX to 9.x or before upgrading to prevent the issue.
- Warning:

- - Ensure all cluster nodes are healthy and in sync before proceeding
  - Once a node is removed from the cluster, it should be powered off or deleted and never reused in the environment.
- Follow the steps with full details at Restore an NSX Manager Node to an Existing NSX Manager Cluster.

1. 1. Take a backup of the NSX Manager. See Backing Up and Restoring NSX Manager.
  2. Download the NSX Manager OVA matching your current version and build. See Broadcom Product Downloads.
  3. Log in to the CLI of a healthy NSX Manager node and identify the UUIDs by running: get nodes.
  4. Identify the API certificate thumbprint by running: get certificate api thumbprint.
  5. Detach the target node from the cluster by running the following command on a different node: detach node #### (where #### is the UUID).
  6. Power off and delete the detached NSX Manager appliance VM from vCenter.
  7. Deploy a new NSX Manager appliance. Ensure the hostname parameter in the OVF deployment is set to the Fully Qualified Domain Name (FQDN).
  8. Join the new node to the cluster:
```
join <existing-manager-ip> cluster-id <cluster-id> username <admin-username> password <admin-password> thumbprint <thumbprint>
```
  9. Verify the cluster is stable using get cluster status before repeating steps 5–8 for the remaining nodes.