Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

To provide a supported method for removing the STS_INTERNAL_SSL_CERT without crashing the VCSA per newly deployed topologies in 6.x ,7.x  and 8.x do not have such STORE.

Symptoms:

• When you delete/Remove STS_INTERNAL_SSL_CERT  the VCenter is not able to start services per STSD service crashes. 
• STS_INTERNAL_SSL_CERT  Store exists  in VCSA's that were  previously Windows  vCenter Servers and then converted to VCSA.  
• The STS_INTERNAL_SSL_CERT must match the Machine SSL Certificate.   
• you may see the following error in the vmware-identity-sts-default.log: 

 

[YYYY-MM-DDTHH:MM:SS] pool-2-thread-5   WARN  com.vmware.identity.interop.ldap.LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: 32

[YYYY-MM-DDTHH:MM:SS] pool-2-thread-5  ERROR com.vmware.identity.interop.ldap.OpenLdapClientLibrary] Exception when calling ldap_search_s: base=cn=2138b306-81d6-4bc6-9212-eea09f2c46e,cn=ServiceRegistrations,cn=LookupService,cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local, scope=2, filter=(objectclass=*), attrs=null, attrsonly=0 com.vmware.identity.interop.ldap.NoSuchObjectLdapException: No such object

• Customers will see the certificate expiry alarm 
  • Alarm name alarm.CertificateStatusAlarm
    [Critical] Alarm alarm.CertificateStatusAlarm on Folder Datacenters
    because Certificate 'C=US,CN=<VC_FQDN>' from 'STS_INTERNAL_SSL_CERT' expires on YY-MM-DD HH:MM:SS.000.

Environment

VMware vCenter Server Appliance 8.0.x

VMware vCenter Server Appliance 7.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x

Cause

STS service is unable to start based on configuration file /usr/lib/vmware-sso/vmware-sts/conf/server.xml and Store STS_INTERNAL_SSL_CERT

Resolution

Remove STS_INTERNAL_SSL_CERT from VECS via shell Script:

1. Take a cold snapshot of all VCSA/PSC's if using Enhanced  Linked Mode (ELM) or a regular snapshot if Standalone VCSA. 
2. Run the lsdoctor option -l to see if there are stale service registrations in the lookup service. If there are no stale registrations, jump to the next step. Otherwise, run python lsdoctor.py -s and python lsdoctor.py -t respectively and continue with the next step. 
3. Download and run the attached script named Delete_ STS_INTERNAL_SSL_CERT.sh
4. Provide permission to the script file with the command chmod +rx Delete_ STS_INTERNAL_SSL_CERT.sh
5. Run script  ./Delete_ STS_INTERNAL_SSL_CERT.sh

• Note: another way of finding out if the STS_INTERNAL_SSL_CERT Store exists is by running  the  command

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

• Note:  You may receive an error when you try to run the script:
  bash:  ./Delete_ STS_INTERNAL_SSL_CERT.sh: /bin/bash^M: bad interpreter: No such file or directory
  This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor.  To resolve this problem, run the following command and rerun the script:
  
  sed -i -e 's/\r$//' Delete_ STS_INTERNAL_SSL_CERT.sh

Remove STS_INTERNAL_SSL_CERT via command line:

• Take a cold snapshot of all VCSA/PSC's if using Enhanced  Linked Mode (ELM) or a regular snapshot if Standalone VCSA
• Run the command from SSH of vCenter server to check if the STS_INTERNAL_SSL_CERT is present in the store : /usr/lib/vmware-vmafd/bin/vecs-cli store list
• Backup /usr/lib/vmware-sso/vmware-sts/conf/server.xml and change "STS_INTERNAL_SSL_CERT" to "MACHINE_SSL_CERT" in this file,
  

  cp /usr/lib/vmware-sso/vmware-sts/conf/server.xml /usr/lib/vmware-sso/vmware-sts/conf/server.xml.old
  
  sed -i 's/STS_INTERNAL_SSL_CERT/MACHINE_SSL_CERT/g' /usr/lib/vmware-sso/vmware-sts/conf/server.xml

• To delete the STS_INTERNAL_SSL_CERT: /usr/lib/vmware-vmafd/bin/vecs-cli store delete --name STS_INTERNAL_SSL_CERT

• Restart all the vCenter services using the command: service-control --stop --all && service-control --start --all;