CCI resource tiles fail with error "401 UNAUTHORIZED "Unauthorized" approximately 1 year after CCI was initially configured

Products

• VCF Automation

Issue/Introduction

• It has been approximately 1 year since cci was initially configured.
• CCI resource tiles fail with error "401 UNAUTHORIZED "Unauthorized" 
• CCI logs show following errors, see additional information section for steps on how to check logfiles:
  
  1 token_review_endpoint.go: Invalid token: failed to validate JWT
1 token_review_endpoint.go:] End: Token review - authenticated = false, user =
1 token_review_endpoint.go: ] Begin: Token review
1 validator.go:] Invalid ID token: failed to verify signature: Could not verify JWT against any configured key.

Environment

Aria Automation 8.18

Cause

The root cause is that the keyset used in the configuration is outdated. Aria Automation 8.18.1 rotates the key pair used to sign the tokens once in a year (every 11 months to be precise) and the old key pair is valid for 30 more days after it's been rotated.

Resolution

To resolve the issue reconfigure the Single Sign-On setup for CCI.

1. Uninstall and unregister the CCI service in vCenter

   • Log in to the vCenter.
   • Under Workload Management, select the Services tab.
   • For the vCenter, select the vCenter that is managing the Supervisor Cluster where the CCI single sign-on service is installed
   • Find the CCI service tile, and Click on ACTIONS dropdown, click Delete
     
     
     
     
     
   • In the popup window, click CONFIRM button in step 1, then click on CONFIRM button in Step 2, then it will start removing the versions from supervisors
     
     
     
     
   • Then in Step 3, click CONFIRM button to delete the service
   • Then the bottom right corner, click DELETE button to finish deletion
     
2. Follow the steps in Setting Up Single Sign-On for CCI  to setup Single Sign-On again for CCI.

 

Additional Information

Steps to check Supervisor cluster logs to match error:

1. ssh into the VCenter as root user 
2. run /usr/lib/vmware-wcp/decryptK8Pwd.py inside the VC as shown in the code block below, it will print out the ip and password of the supervisor cluster.

ssh root@###### VMware vCenter Server 8.0.2.00000 Type: vCenter Server with an embedded Platform Services Controller (root@######) Password: Last login: Wed Jan 17 22:31:00 2024 from ###### Connected to service * List APIs: "help api list" * List Plugins: "help pi list" * Launch BASH: "shell" Command> shell Shell access is granted to root root@###### [ ~ ]# /usr/lib/vmware-wcp/decryptK8Pwd.py Read key from file Connected to PSQL Cluster: domain-######:###### IP: <IP of Supervisor Cluster> PWD: <Password of Supervisor Cluster> ------------------------------------------------------------


3. Use the ip (<IP of Supervisor Cluster>) and the password (<Password of Supervisor Cluster>) found in the previous step, ssh into the supervisor cluster, then run the cmd in the below block:


4. CCI Supervisor Service pod is under a namespace with "cci-service" as part of its name, hence we can use it to locate the namespace,

kubectl --kubeconfig /etc/kubernetes/admin.conf get ns | grep cci-service

svc-cci-service-domain-### Active 23m

5. List the pods in this namespace, you will find a pod with the prefix "cci-service", 

kubectl --kubeconfig /etc/kubernetes/admin.conf -n svc-cci-service-domain-###get pod

NAME READY STATUS RESTARTS AGE
cci-service-##### 1/1 Running 0 25m


6. tail the logs of the pod, you will find the logging of each token request and the result of the token request

 kubectl --kubeconfig /etc/kubernetes/admin.conf -n svc-cci-service-domain-### logs -f cci-service-#####