Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

To provide a supported method for removing the STS_INTERNAL_SSL_CERT without crashing the VCSA ; as newly deployed topologies in vCSA 6.x,7.x and 8.x do not have such STORE.

• When you delete/remove STS_INTERNAL_SSL_CERT , the vCenter is not able to start all services as stsd service crashes. 
• STS_INTERNAL_SSL_CERT  store exists  in VCSA's that were  previously Windows  vCenter Servers and then converted to VCSA.  
• The STS_INTERNAL_SSL_CERT must match the Machine SSL Certificate.
  
  
• You may see the following errors in the log file /var/log/vmware/sso/vmware-identity-sts-default.log:


  [YYYY-MM-DDTHH:MM:SS] pool-2-thread-5   WARN  com.vmware.identity.interop.ldap.LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: 32
  [YYYY-MM-DDTHH:MM:SS] pool-2-thread-5  ERROR com.vmware.identity.interop.ldap.OpenLdapClientLibrary] Exception when calling ldap_search_s: base=cn=2138b306-81d6-####-9212-#######,cn=ServiceRegistrations,cn=LookupService,cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local, scope=2, filter=(objectclass=*), attrs=null, attrsonly=0 com.vmware.identity.interop.ldap.NoSuchObjectLdapException: No such object

  


• You may see the following errors in the log file /var/log/vmware/sso/tomcat/catalina.log:
  

  [YYYY-MM-DDTHH:MM:SS]SEVE org.apache.catalina.core.StandardService Failed to initialize connector [Connector[com.vmware.identity.tomcat.VECSAwareHttp11NioProtocol-7444]]
  org.apache.catalina.LifecycleException: Protocol handler initialization failed
  Caused by: java.lang.IllegalArgumentException: Native platform error [code: 4312][Native platform error [code: 4312][Opening store 'STS_INTERNAL_SSL_CERT' failed. [Server: __localhost__, User: __localuser__]]]

• Customers will see the certificate expiry alarm 
  • Alarm name alarm.CertificateStatusAlarm
    [Critical] Alarm alarm.CertificateStatusAlarm on Folder Datacenters because Certificate 'C=US,CN=<VC_FQDN>' from 'STS_INTERNAL_SSL_CERT' expires on YY-MM-DD HH:MM:SS.000.

Environment

VMware vCenter Server Appliance 8.0.x

VMware vCenter Server Appliance 7.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x

Cause

STS service is unable to start based on configuration file /usr/lib/vmware-sso/vmware-sts/conf/server.xml and store STS_INTERNAL_SSL_CERT

Resolution

1. Scripted way to remove STS_INTERNAL_SSL_CERT from VECS via shell Script:

1. Take a cold snapshot of all VCSA/PSC's (mandatorily) if using Enhanced  Linked Mode (ELM) or a regular snapshot if standalone vCSA. 
2. Download the lsdoctor tool and run python lsdoctor.py-l to see if there are stale service registrations in the lookup service. If there are no stale registrations, jump to the next step. Otherwise, run python lsdoctor.py -s and python lsdoctor.py -t respectively and continue with the next step. 
3. Download the attached script named Delete_ STS_INTERNAL_SSL_CERT.sh
4. Provide permission to the script file with the command chmod +rx Delete_ STS_INTERNAL_SSL_CERT.sh
5. Run script  ./Delete_ STS_INTERNAL_SSL_CERT.sh

• Note: another way of finding out if the STS_INTERNAL_SSL_CERT Store exists is by running  the  command :
  

  for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

• NOTE :  You may receive an error when you try to run the script:
  bash:  ./Delete_ STS_INTERNAL_SSL_CERT.sh: /bin/bash^M: bad interpreter: No such file or directory
  This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor.  To resolve this problem, run the following command and rerun the script:
  
  sed -i -e 's/\r$//' Delete_ STS_INTERNAL_SSL_CERT.sh



2. Manual way to remove STS_INTERNAL_SSL_CERT via command line:

• Take a cold snapshot of all VCSA/PSC's (mandatorily)if using Enhanced  Linked Mode (ELM) or a regular snapshot if standalone vCSA.
• Run the command from SSH of vCSA to check if the STS_INTERNAL_SSL_CERT is present in the store : /usr/lib/vmware-vmafd/bin/vecs-cli store list
• Backup /usr/lib/vmware-sso/vmware-sts/conf/server.xml and change "STS_INTERNAL_SSL_CERT" to "MACHINE_SSL_CERT" in this file,
  

  cp /usr/lib/vmware-sso/vmware-sts/conf/server.xml /usr/lib/vmware-sso/vmware-sts/conf/server.xml.old
  
  sed -i 's/STS_INTERNAL_SSL_CERT/MACHINE_SSL_CERT/g' /usr/lib/vmware-sso/vmware-sts/conf/server.xml

• To delete the STS_INTERNAL_SSL_CERT: /usr/lib/vmware-vmafd/bin/vecs-cli store delete --name STS_INTERNAL_SSL_CERT

• Restart all the vCenter services using the command: service-control --stop --all && service-control --start --all;

Additional Information

Using the 'lsdoctor' Tool
Manually reviewing certificates in VMware Endpoint Certificate Store