NSX Load Balancer Certificate Rotation fails for VKS due to special characters present in vCenter TRUSTED_ROOTS certificate

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• After following the correct procedure to rotate the NSX Load Balancer certificate for VKS, the new certificate is not seen on NSX Manager 
• The new certificate for the load balancer is seen and updated in the vCenter GUI 
• Supervisor is stuck in configuring state 
• wcpsvc.log on vCenter Server shows: 
  "DefaultMessage": "System error occurred on control plane<control_plane_node>. Details Failed to validate certificate in vc_trust_bundle, stopped syncing: 'ascii' codec can't encode character '\\xe4' in position 257: ordinal not in range(128)", "Args": ["<control_plane_node>", "Failed to validate certificate in vc_trust_bundle, stopped syncing: 'ascii' codec can't encode character '\\xe4' in position 257: ordinal not in range(128)"]}}]}}]
  • \\xe4 can be any other character which cannot be encoded 
  • In this case it was ä but other special characters may face same issue 

Environment

• VMware vCenter Server 8.x 
• VMware Kubernetes Service (VKS) 
• VMware NSX

Cause

One of the certificates sent from the vCenter Server in VC_TRUST_BUNDLE contains a special character which cannot be ASCII encoded and therefore this blocks the new load balancer certificate from getting pushed down also. 

Resolution

1. Verify the certificate entries in VECS on the vCenter Server 
   • Check all the subject, SAN, DNS entries, etc. for any special characters 
   • Remove the certificate if unused or renew with updated fields via the vCert script 
     
     
2. If no obvious entries are found in VECS on the vCenter Server, locate VC_TRUST_BUNDLE in /var/lib/node.cfg
   • This will provide you with the base64 encoded output of the certificates being sent
   • echo "<full_base64_encoded_output_from_VC_TRUST_BUNDLE_in_/var/lib/node.cfg"> | base64 -d
   • This will give you the decoded output for the certificates where you can then check for the special character present 
   • It may be a TRUSTED_ROOT certificate on the vCenter Server but the field containing the special character is not shown in VECS 
   • Remove/renew the culprit certificate via the vCert script