Not allowed to apply a service-type API to a CA-signed certificate without hostname check passing

Products

• VMware NSX

Issue/Introduction

• After uploading a new CA-signed certificate to NSX, when attempting to apply the certificate to the API service type, there is an error:
  "Not allowed to apply a service-type API to a CA-signed certificate without hostname check passing."
• Error will show when attempting apply the certificate using the NSX UI or API 
• When running the API Post command POST /api/v1/trust-management/certificates/<cert-id>?action=apply_certificate&service_type=API&node_id=<node-id> the message shows:
  "error_message": "Not allowed to apply a service-type API to a CA-signed certificate without hostname check passing."
• When applying the certificate using the UI (in NSX 4.2.x), when selecting Service/Entity "API" and choosing any Node, the message shows:
  "Error: Not allowed to apply a service-type API to a CA-signed certificate without hostname check passing. (Error code: 5158)"
• When running the below command on any NSX managers, the result is empty (nothing is returned):

  • /opt/vmware/nsx-common/python/nsx_utils/ip_utils.py -dx <NSX-Manager-IP>

Environment

• NSX 4.1.x and above

Cause

• The error is shown when hostnames and/or FQDN in the "Subject Alternative Name" entity of the certificate do not match the DNS records of the NSX Managers and/or VIP hostnames
• However this error will also show if the hostnames from the certificate and the DNS records have a case mismatch
• For example, reverse DNS entries may have the hostname shown in upper case, and the certificate shows the hostnames with only lower case
• Originally configured DNS servers on the managers no longer have a reverse lookup for the manager IPs after environment change.

Resolution

• Create A or AAAA (if using IPv6) and PTR (reversed) DNS entries for the NSX Managers and VIP fully qualified domain names (FQDN).
  • Please ensure the records are created under the correct domain root if there are multiple domains used in the environment.  
  • See Configure NSX Manager for access by DNS server in the install guide for further assistance.
• Ensure DNS records of the NSX Managers and VIP hostnames match the FQDN entries for the CA-signed certificate
• As a workaround for a case mismatch between the certificate fields and the DNS records either
  • Edit the reverse DNS records to match the case of the fully qualified domain names (FQDN)
  • Regenerate the CA-signed certificate with fully qualified domain names (FQDN) matching the case of the DNS records
• If the DNS server has been changed, the new valid DNS can be added by using set name-servers <IP-ADDRESS> on each of the managers to point to the correct DNS server with the reverse lookup resolvable.