"KeyStores with multiple certificates are not supported on the base class", Update Manager Service (vmware-updatemgr) crashes immediately with core.worker.XXX dump

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Update Manager Service stops immediately with core dump (/storage/core/core.worker.XXX)
• You will see entries similar to below in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log

[2021-05-12 11:27:43:247 'managedJetty' 140186298865408 INFO] [managedJetty, 453] Jetty: Caused by: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.
[2021-05-12 11:27:43:253 'managedJetty' 140186298865408 INFO] [managedJetty, 449] Jetty: SslContextFactory$Client instead)
[2021-05-12 11:27:43:253 'managedJetty' 140186298865408 INFO] [managedJetty, 449] Jetty:      at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275)
[2021-05-12 11:27:43:253 'managedJetty' 140186298865408 INFO] [managedJetty, 449] Jetty:      at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256)
[2021-05-12 11:27:43:253 'managedJetty' 140186298865408 INFO] [managedJetty, 449] Jetty:      at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374)
[2021-05-12 11:27:43:253 'managedJetty' 140186298865408 INFO] [managedJetty, 449] Jetty:      at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
[2021-05-12 11:27:43:253 'managedJetty' 140186298865408 INFO] [managedJetty, 449] Jetty:      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
[2021-05-12 11:27:43:589 'managedJetty' 140186954495744 INFO] [managedJetty, 466] Jetty terminated.
[2021-05-12 11:27:43:639 'managedJetty' 140186954495744 ERROR] [managedJetty, 502] Jetty unexpectedly stopped with status 254. Starting new one.
[2021-05-12 11:27:43:639 'managedJetty' 140186954495744 ERROR] [managedJetty, 529] Jetty server keeps crashing! Giving up.

• Core dumps (sample naming below) are created in /storage/core for every service restart:

-rw-rw-r-- 1 svc.datamover support 195M May 12 09:41 core.worker.10382
-rw-rw-r-- 1 svc.datamover support 201M May 12 09:41 core.worker.11185
-rw-rw-r-- 1 svc.datamover support 194M May 12 09:41 core.worker.11689
-rw-rw-r-- 1 svc.datamover support 194M May 12 09:41 core.worker.12585
-rw-rw-r-- 1 svc.datamover support 194M May 12 09:41 core.worker.19495

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.x

Cause

This issue is seen due to missing keyword ($Server) in /usr/lib/vmware-updatemgr/bin/jetty-vum-ssl.xml.

Resolution

To workaround this issue, please follow below steps :

• Connect to VCSA using SSH (Putty)
• Verify the file /usr/lib/vmware-updatemgr/bin/jetty-vum-ssl.xml and check for the line mentioned below for missing $Server entry : 
  • less /usr/lib/vmware-updatemgr/bin/jetty-vum-ssl.xml | grep -i "org.eclipse.jetty.util.ssl.SslContextFactory"

<Configure class="org.eclipse.jetty.util.ssl.SslContextFactory" id="sslContextFactory">

• Sample result :

Reference:
           <?xml version="1.0"?>
           <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_3.dtd">
           <Configure class="org.eclipse.jetty.util.ssl.SslContextFactory" id="sslContextFactory">
           <Set name="KeyStorePath">
            <Property default="." name="jetty.home"/>/../ssl/vmware-vum.keystore</Set>
            <Set name="KeyStorePassword">OBF:1wn91zev1xtd1y7z1xu51zef1wmx
            </Set>

• If the variable '$Server' is missing in the above mentioned line, edit the file to add the $Server entry like below

<Configure class="org.eclipse.jetty.util.ssl.SslContextFactory$Server" id="sslContextFactory">
<Set name="KeyStorePath">

• Make sure the file looks as below, you may edit the file using 'vi' editor
  • vi /usr/lib/vmware-updatemgr/bin/jetty-vum-ssl.xml
  • Press 'Insert' Key
  • Add the $Server entry
  • Edit the edit with keys Esc, followed by wq! and Enter

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_3.dtd">
<Configure class="org.eclipse.jetty.util.ssl.SslContextFactory$Server" id="sslContextFactory">
<Set name="KeyStorePath">
<Property default="." name="jetty.home"/>/../ssl/vmware-vum.keystore</Set>

• Restart the update manager service.
  • service-control --stop vmware-updatemgr && service-control --start vmware-updatemgr