Sync and delete Content Library failed

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

• Sync or delete Content Library failed with "RuntimeFault.summary" 
  
• When trying to deploy new virtual machine using template in Content Library, process failed with "Unable to process template"

Environment

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Cause

1. There are duplicate entries in Content Library certificate database.
2. The issue can be identified following these steps:
3.  Below log messages are observed in /var/log/vmware/content-library/cls.log of the vCenter
   
   

   #less /var/log/vmware/content-library/cls.log
   ####-##-##T##:##:##.###+##:## | ERROR    | l#####m-######-auto-####-h5:70####91-a2 | cls-simple-activity-9     | VdcsFutureResultCallback       | Activity threw exception.
   
   com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
   
       messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
   
       id = vapi.bindings.method.impl.unexpected,
   
       defaultMessage = Provider method implementation threw unexpected exception: result returns more than one elements; nested exception is javax.persistence.NonUniqueResultException: result returns more than one elements,
   
       args = [result returns more than one elements; nested exception is javax.persistence.NonUniqueResultException: result returns more than one elements],
   
       params = <null>,
   
       localized = <null>
   
   }],
   
       data = <null>,
   
       errorType = INTERNAL_SERVER_ERROR
   
   }

   
   or 
   
   

   Error:InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
   messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
   id = vapi.bindings.method.impl.unexpected,
   defaultMessage = Provider method implementation threw unexpected exception: (vim.fault.CannotDeleteFile) {
   faultCause = null,
   faultMessage = null,
   file = [cs4-83-iaas-prod-c1-vsan] /contentlib-89d21772-###-####-####-fb67f75894bd/0b9a35fc-####-####-####-87c749fb63dc
   },
   args = [(vim.fault.CannotDeleteFile) {
   faultCause = null,
   faultMessage = null,

   
   
   
4. When checking Content Library certificate items in vCenter database, duplicate entries were found:
5. Log in to vCenter via ssh with root account
6. Log in to VCDB with below command:
   
   

   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

   
   
   
7. Check Content Library certificate items with command:
   
   
   

   SELECT * FROM cl_libraryitem_certificate_verification;

(For example, below screenshot shows there are 3 pairs of same itemid, this means there are 3 duplicate entries in the database. )

Resolution

      Delete the duplicate entries in the Content Library certificate database. Follow the steps below:

1. Make sure a valid backup or snapshot exists for vCenter before carrying out the action plan. If the vCenter is part of Enhanced Linked mode, please take offline snapshots for all vCenters in Enhanced Linked mode.
2. Log in to vCenter vis ssh with root account
3. Log in to VCDB using below command:
   
   

   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

   
   
   
4. Delete duplicate entries in the database using below command:
   
   

   DELETE FROM cl_libraryitem_certificate_verification cert_ver_old USING (
                           SELECT MIN(ctid) as ctid, itemId
                           FROM cl_libraryitem_certificate_verification
                           GROUP BY itemId HAVING COUNT(*) > 1
                   ) cert_ver_new
                   WHERE cert_ver_new.itemId = cert_ver_old.itemId
                   AND cert_ver_new.ctid <> cert_ver_old.ctid;