TKGi rotate-certificates Fails with "Unknown Error Occurred Rotating NSX Certs" | "Hostname Not Verified"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

When attempting to rotate cluster certificates using the command tkgi rotate-certificates <cluster> --all, the Last Action State of the cluster shows "failed", and the PKS API logs contain the following error:

"Unknown error occurred rotating nsx certs"

"Hostname <hostname> not verified"

This issue prevents successful certificate rotation.

Environment

VMware Tanzu Kubernetes Grid Integrated Edition

Cause

This error occurs due to an invalid NSX Manager CA certificate. The CA certificate is configured in two locations within the TKGi tile:

1. NSX Manager CA in TKGi Tile
   • CredHub Path: .properties.network_selector.nsx.nsx-t-ca-cert
   • UI Path: TKGi Tile > Networking > NSX Manager CA Cert
2. NSX Manager CA in BOSH Tile
   • CredHub Path: .iaas_configuration.nsx_ca_certificate
   • UI Path: BOSH Tile > vCenter Config > NSX CA Cert

Resolution

Option 1: Fix the Certificate Issue

To resolve the error, ensure that the correct and valid NSX Manager CA certificate is configured in both the TKGi and BOSH tiles.

• Verify the NSX Manager CA Cert value in the TKGi tile and BOSH tile.
• If the certificate is incorrect or expired, update it with a valid CA certificate.
• After correcting the certificate, apply changes and retry the certificate rotation.

Option 2: Skip NSX Certificate Rotation

If the NSX-T certificates have not expired, you can skip their rotation while rotating other cluster certificates by running the rotation command with the --skip-nsx flag.

Option 3: Adjust NSX Manager CA Certificate Settings to Prevent Rotation Errors

In TKGi Documentation, the NSX Manager CA Cert field and the Disable SSL Certificate Verification option are intended to be mutually exclusive. Incorrect configuration can lead to the "Hostname not verified" error during certificate rotation.

Key Configuration Rules:

1. If SSL certificate verification is disabled, the NSX Manager CA Cert field must be left blank.
2. If a certificate is entered in the NSX Manager CA Cert field, SSL certificate verification must remain enabled.
3. If both a certificate is entered and SSL certificate verification is disabled, insecure mode takes precedence, which can cause conflicts.

How to Avoid the Error During Certificate Rotation:

• If SSL verification is enabled but the rotation fails, ensure the correct NSX Manager CA certificate is entered.
• If SSL verification is disabled, remove the certificate from the NSX Manager CA Cert field to prevent conflicts.

Following these guidelines ensures TKGi behaves as expected during certificate rotation and prevents "Hostname not verified" errors.

Additional Information

Configure Tanzu Kubernetes Grid Integrated Edition | Networking