Subscribed Content Library failing to sync with error "INVALID_ELEMENT

Products

VMware vCenter Server

Issue/Introduction

Address subscribed content library service sync issue due to certificate not matching the com.vmware.cl extension

Symptoms:
If subscribed content library receiving INVALID_ELEMENT_TYPE in /var/log/vmware/content-library/cls.log

2023-11-13T20:38:35.591Z | DEBUG    | q-138345:h5ui-getProperties:urn:vapi:com.vmware.content.Library:#######-####-####-####-############:########-####-####-####-d00afd27b464:1855923959:ContentLibrarySpecificCapabilitiesPropertyProviderAdapter:647623-e321-h5:70153732 | tomcat-http-21            | LocalProvider                  | call to invoke() for service 'com.vmware.content.library.subscriptions', operation 'get'
2023-11-13T20:38:35.599Z | DEBUG    | q-138345:h5ui-getProperties:urn:vapi:com.vmware.content.Library:#######-####-####-####-############:########-####-####-####-d00afd27b464:1855923959:ContentLibrarySpecificCapabilitiesPropertyProviderAdapter:647623-e321-h5:70153732 | tomcat-http-21            | ApiMethodSkeleton              | Method com.vmware.content.library.subscriptions.get threw an exception
com.vmware.vapi.std.errors.InvalidElementType: InvalidElementType (com.vmware.vapi.std.errors.invalid_element_type) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vdcs.cls-main.library_subscribed,
    defaultMessage = Library #######-####-####-####-############ is subscribed.,
    args = [#######-####-####-####-############],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = INVALID_ELEMENT_TYPE
}
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_362]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_362]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_362]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_362]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_362]
    at com.vmware.vcde.util.vapi.VapiErrorBuilder.build(VapiErrorBuilder.java:70) ~[vsphere-util-lib-1.0.0.jar:?]
    at com.vmware.vcde.util.vapi.VapiErrorBuilder.build(VapiErrorBuilder.java:90) ~[vsphere-util-lib-1.0.0.jar:?]
    at com.vmware.cl.validators.LibraryValidator.checkLibraryIsNotSubscribed(LibraryValidator.java:124) ~[cls-main-1.0.0.jar:?]
    at com.vmware.cl.vapi.SubscriptionsImpl.validateLibrary(SubscriptionsImpl.java:105) ~[cls-vmodl-impl-1.0.0.jar:?]
    at com.vmware.cl.vapi.SubscriptionsImpl.get(SubscriptionsImpl.java:95) ~[cls-vmodl-impl-1.0.0.jar:?]
    at com.vmware.content.library.SubscriptionsApiInterface$GetApiMethod.doInvoke(SubscriptionsApiInterface.java:145) ~[cls-vmodl-1.0.0.jar:?]
    at com.vmware.vapi.internal.bindings.ApiMethodSkeleton.invoke(ApiMethodSkeleton.java:233) [vapi-runtime-2.100.0.jar:?]
    at com.vmware.vapi.provider.ApiMethodBasedApiInterface.invoke(ApiMethodBasedApiInterface.java:86) [vapi-runtime-2.100.0.jar:?]

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0
VMware vCenter Server 6.x

Cause

You will see similar certificate error in /var/log/vmware/content-library/cls.log

2023-11-13T20:43:18.574Z | ERROR | lon1bh5h-658376-auto-e409-h5:70153882 | tomcat-http-19 | ThumbprintTrustStrategy | SSL thumbprint mismatch: Received ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##, expected
2023-11-13T20:43:18.575Z | ERROR | lon1bh5h-658376-auto-e409-h5:70153882 | tomcat-http-19 | VcspClientImpl | Remote library certificate error: certificate_unknown(46)
org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
   at org.bouncycastle.jsse.provider.ProvSSLSocketWrap.checkServerTrusted(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.jsse.provider.ProvTlsClient$1.notifyServerCertificate(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.tls.TlsUtils.processServerCertificate(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.tls.TlsClientProtocol.handleServerCertificate(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.tls.TlsClientProtocol.handleHandshakeMessage(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.tls.TlsProtocol.processHandshakeQueue(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.tls.TlsProtocol.processRecord(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.tls.RecordStream.readRecord(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.tls.TlsProtocol.safeReadRecord(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.tls.TlsProtocol.blockForHandshake(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]
   at org.bouncycastle.tls.TlsClientProtocol.connect(Unknown Source) ~[bctls-fips-1.0.10.jar:1.0.10]

Resolution

1. Connect to vCenter Appliance with SSH client via root account
2., If you are at Command> Then type in shell to get to the bash prompt. Otherwise skip this step
3. We will need to create a folder to store the solutions user certificate and key

mkdir /certificate

4. To export vpxd-extension cert and key to certificate folder run the commands below

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

5. Run the command below to update com.vmware.cl extension certificate

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.cl -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <vCenter-FQDN> -u [email protected]

Press enter and then enter your password for the sso account

Note: If SSO was changed from the default vsphere.local, make sure you update the -u portion of the command to reflect correct local SSO domain

6. Restart the vmware-content-library service

service-control --restart vmware-content-library

7. Test to confirm subscribed content library can sync correctly

Additional Information

Impact/Risks:
It will cause Subscribed Content Libraries sync to fail

Also referenced - EAM may be impacted EAM "Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password" after replacing the vCenter Server certificates