Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH for vSphere 9.X

Products

• VMware vCenter Server

Issue/Introduction

The STS_INTERNAL_SSL_CERT is a legacy certificate store in the VMware Endpoint Certificate Store (VECS) of a vCenter Server Appliance (VCSA).
It typically exists on vCenter instances, that were originally deployed as version 5.5 or 6.0 and subsequently upgraded.
Manual removal of said store is not recommended as it can impact the vCenter. 

Similar scenario as in KB https://knowledge.broadcom.com/external/article?articleNumber=316625, however the process is changed for vSphere 9.X.
In vCenter 9.0+, the STS service moved to an embedded Tomcat instance.

Due to this change, the SSL certificate store configurations are no longer defined in /usr/lib/vmware-sso/vmware-sts/conf/server.xml, and are instead defined in /usr/lib/vmware-sso/vmware-sts/conf/catalina.properties.

The format of the file is different, it's no longer an XML file, but a standard Java properties file.

Environment

vSphere 9.X

Cause

STS service is unable to start based on configuration file /usr/lib/vmware-sso/vmware-sts/conf/catalina.properties and store STS_INTERNAL_SSL_CERT



Resolution

A. Manual Method
Remediation Steps for vCenter 9.0

• 1. Take a snapshot
  Take a cold snapshot of all VCSA/PSC's (mandatory) if using Enhanced Linked Mode (ELM), or a regular snapshot if standalone vCSA.
  For reference ELM is deprecated in 9.X but can still be in use
  
  
• 2. Verify the store exists.
  Run the command from an SSH session on the vCenter to check if the STS_INTERNAL_SSL_CERT is present in the store:

  # /usr/lib/vmware-vmafd/bin/vecs-cli store list
  MACHINE_SSL_CERT
  TRUSTED_ROOTS
  TRUSTED_ROOT_CRLS
  machine
  vsphere-webclient
  vpxd
  vpxd-extension
  SMS
  STS_INTERNAL_SSL_CERT                                       => this is a Stale entry from 5.X/ 6.X
  APPLMGMT_PASSWORD
  data-encipherment
  BACKUP_STORE
  hvc
  wcpsvc

  


• 3. Backup and update catalina.properties.
  Backup /usr/lib/vmware-sso/vmware-sts/conf/catalina.properties, and change all occurrences of STS_INTERNAL_SSL_CERT to MACHINE_SSL_CERT in this file:
  

  # cp /usr/lib/vmware-sso/vmware-sts/conf/catalina.properties /usr/lib/vmware-sso/vmware-sts/conf/catalina.properties.old
  
  # sed -i 's/STS_INTERNAL_SSL_CERT/MACHINE_SSL_CERT/g' /usr/lib/vmware-sso/vmware-sts/conf/catalina.properties

• 4. Delete the store from VECS.
  To delete the STS_INTERNAL_SSL_CERT store:
  

  # /usr/lib/vmware-vmafd/bin/vecs-cli store delete --name STS_INTERNAL_SSL_CERT

• 5. Restart all vCenter services.
  Restart all the vCenter services using the command:
  

  # service-control --stop --all && service-control --start --all

B. Scripted Method
Remediation Steps for vCenter 9.0

• 1. Take a snapshot
  Take a cold snapshot of all VCSA/PSC's (mandatory) if using Enhanced Linked Mode (ELM), or a regular snapshot if standalone vCSA.
  For reference ELM is deprecated in 9.X but can still be in use
  
  
• 2. Verify the store exists.
  Run the command from an SSH session on the vCenter to check if the STS_INTERNAL_SSL_CERT is present in the store:

  # /usr/lib/vmware-vmafd/bin/vecs-cli store list
  MACHINE_SSL_CERT
  TRUSTED_ROOTS
  TRUSTED_ROOT_CRLS
  machine
  vsphere-webclient
  vpxd
  vpxd-extension
  SMS
  STS_INTERNAL_SSL_CERT                                       => this is a Stale entry from 5.X/ 6.X
  APPLMGMT_PASSWORD
  data-encipherment
  BACKUP_STORE
  hvc
  wcpsvc

• 3. Download the attached script named Delete_STS_INTERNAL_SSL_CERT_9X.sh

• 4. Provide permission to the script file with the command

  # chmod +rx Delete_STS_INTERNAL_SSL_CERT_9X.sh

• 5. Run script 
  

  # ./Delete_STS_INTERNAL_SSL_CERT.sh

• 6. Script does the exact same steps, including restart of all services  

Additional Information

The store STS_INTERNAL_SSL_CERT contains a certificate with the Alias __MACHINE_CERT, same as store MACHINE_SSL_CERT.
With that in mind, said store does not get updated when replacing certificates via certificate-manager or vCert, so it's not the current  __MACHINE_CERT.
As a side effect, it can also trigger the certificate alarm, while it will not impact Vcenter availability, triggering said alarm is justified as there is a store with an expiring certificate. 

For 8.X vCenter please use https://knowledge.broadcom.com/external/article?articleNumber=316625