"FATA[0004] Failed to get available workloads, response from the server was invalid." error while attempting to login to workload clusters from jumpbox

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• Attempting to login to the guest cluster using "kubectl-vsphere login", following error is seen: "FATA[0004] Failed to get available workloads, response from the server was invalid."
• The ncp and vsphere-csi-controller pods are found to be in a CrashLoopBackOff state.
• The following errors are seen in nsx-ncp pod logs:
  ncp MainThread W] nsx_ujo.ncp.vc.session Failed to get JWT token: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007), will retry after 120 seconds

Environment

vSphere with Tanzu

Cause

NSX certificates are expired 

Resolution

1. Regenerate the NSX certificates. Refer Renew or replace the self-signed SSL certificates assigned to various components of NSX version 4.2 and later through the GUI interface Only.
2. Restart the Cert Manager 'cainjector' Deployment by running the following command on the supervisor control plane node:
   kubectl rollout restart deployment -n vmware-system-cert-manager cert-manager-cainjector