Sync and delete Content Library failed

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

Symptoms:

• Sync or delete Content Library failed with "RuntimeFault.summary" 
  
• When trying to deploy new virtual machine using template in Content Library, process failed with "Unable to process template"

Environment

VMware vCenter Server 7.0

VMware vCenter Server 8.0

Cause

There are duplicate entries in Content Library certificate database.

 

The issue can be identified following these steps:

 

1 Below log messages are observed in /var/log/vmware/content-library/cls.log:

2024-08-21T18:24:42.666+08:00 | ERROR    | lnemua7m-941673-auto-k6lm-h5:70154891-a2 | cls-simple-activity-9     | VdcsFutureResultCallback       | Activity threw exception.

com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {

    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {

    id = vapi.bindings.method.impl.unexpected,

    defaultMessage = Provider method implementation threw unexpected exception: result returns more than one elements; nested exception is javax.persistence.NonUniqueResultException: result returns more than one elements,

    args = [result returns more than one elements; nested exception is javax.persistence.NonUniqueResultException: result returns more than one elements],

    params = <null>,

    localized = <null>

}],

    data = <null>,

    errorType = INTERNAL_SERVER_ERROR

}

 

2 When checking Content Library certificate items in vCenter database, duplicate entries were found:

1. login to vCenter via ssh with root account
2. login to VCDB with below command:
   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
3. check Content Library certificate items with command:
   SELECT * FROM cl_libraryitem_certificate_verification;

(For example, below screenshot shows there are 3 pairs of same itemid, this means there are 3 duplicate entries in the database. )

 

Resolution

Delete the duplicate entries in Content Library certificate database.

Make sure a valid backup or snapshot exists for vCenter before carrying out the action plan. If the vCenter is part of Enhanced Linked mode, please take offline snapshots for all vCenters in Enhanced Linked mode.

1. login to vCenter vis ssh with root account
2. login to VCDB using below command:
   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
3. delete duplicate entries in database using command:
   DELETE FROM cl_libraryitem_certificate_verification cert_ver_old USING (
                        SELECT MIN(ctid) as ctid, itemId
                        FROM cl_libraryitem_certificate_verification
                        GROUP BY itemId HAVING COUNT(*) > 1
                ) cert_ver_new
                WHERE cert_ver_new.itemId = cert_ver_old.itemId
                AND cert_ver_new.ctid <> cert_ver_old.ctid;