Not allowed to apply a service-type API to a CA-signed certificate without hostname check passing

Products

• VMware NSX

Issue/Introduction

• After uploading a new CA-signed certificate to NSX, when attempting to apply the certificate to the API service type, there is an error:
  "Not allowed to apply a service-type API to a CA-signed certificate without hostname check passing."
• Error will show when attempting apply the certificate using the NSX UI or API 
• When running the API Post command POST /api/v1/trust-management/certificates/<cert-id>?action=apply_certificate&service_type=API&node_id=<node-id> the message shows:
  "error_message": "Not allowed to apply a service-type API to a CA-signed certificate without hostname check passing."
• When applying the certificate using the UI (in NSX 4.2.x), when selecting Service/Entity "API" and choosing any Node, the message shows:
  "Error: Not allowed to apply a service-type API to a CA-signed certificate without hostname check passing. (Error code: 5158)"
  
  

Environment

• NSX 4.1.x and above

Cause

• The error is shown when hostnames and/or FQDN in the "Subject Alternative Name" entity of the certificate do not match the DNS records of the NSX Managers and/or VIP hostnames
• However this error will also show if the hostnames from the certificate and the DNS records have a case mismatch
• For example, reverse DNS entries may have the hostname shown in upper case, and the certificate shows the hostnames with only lower case

Resolution

• Create forward and reverse DNS entries for the NSX Managers and VIP fully qualified domain names (FQDN).
  • See Configure NSX Manager for access by DNS server in the install guide for further assistance.
• Ensure DNS records of the NSX Managers and VIP hostnames match the FQDN entries for the CA-signed certificate
• As a workaround for a case mismatch between the certificate fields and the DNS records either
  • Edit the reverse DNS records to match the case of the fully qualified domain names (FQDN)
  • Regenerate the CA-signed certificate with fully qualified domain names (FQDN) matching the case of the DNS records