Manually remove a Content Library from vCenter Server Appliance when the datastore it resided on is no longer accessible or the Content Library has been corrupted

Products

• VMware vCenter Server

Issue/Introduction

• A Content Library cannot be deleted from the vSphere Web UI.
  
  
• When attempting to delete the Content Library using the vSphere Web UI, the task fails.
  
  Error:
A general system error occurred:
Cannot delete content from library 19b1fa06-####-####-####-23379ec0e9c7, reason: (vim.fault.InaccessibleDatastore) {
faultCause = null,
faultMessage = null,
datastore = ManagedObjectReference: type = Datastore, value = datastore-####, serverGuid = 3c723c9d-####-####-####-b24fb5a070c0,
name = <DS_Name>,
detail = notAccessibleWithPrivilege
}.


• The backing datastore for the Content Library is no longer accessible after the datastore has been unmounted and/or removed from hosts.
  
  
• The Content Library was corrupted from accidental deletion of the Content Library items from disk or due to a corrupted content library item uploaded to the library resulting in new items failing to deploy from the Content Library.
• Some Library items fails to delete and remain as stale.
  
  /var/log/vmware-content-library/cls.log
  
  YYYY-MM-DDTHH:MM:SS409Z | ERROR    | cls-simple-activity-3     | DeleteLibraryActivity          | Library items deletion of library 19b1fa06-####-####-####-23379ec0e9c7 failed at library item 26a5342c-####-####-####-28c539b3f444.
  YYYY-MM-DDTHH:MM:SS752Z | ERROR    | cls-simple-activity-4     | DeleteLibraryActivity          | Library items deletion of library 19b1fa06-####-####-####-23379ec0e9c7 failed.

Environment

VMware vCenter Server

Cause

• Part of the deletion process for a Content Library is cleaning up the Content Library files which reside on a VMFS datastore. If this part of the workflow fails, the deletion process can run into an error and the Content Library fails to delete completely. The CL may still show content with size and details, but the data does not exist on disk.

Resolution

Note: The steps below are irreversible. Before you make those changes, ensure that you have created a fresh backup or snapshot of the vCenter Server. If the vCenter Server is a member of an Enhanced Linked Mode (ELM) replication setup, keep in mind that you need to create backups or offline snapshots (in powered off state) for all ELM nodes. Similarly, when you need to restore the backup of this vCenter, or roll back to the snapshot, all of the other ELM nodes need to be restored/rolled back as well. For more information, see https://knowledge.broadcom.com/external/article/313886.

• Stop the vpxd and content library services:
  
  service-control --stop vmware-vpxd

service-control --stop vmware-content-library

• To remove the Content Library from vCenter database: 

1. 1. Connect to the vCenter Database using to psql tool in the SSH:
      
      /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
      
      
   2. In psql run the following query to find the ID of the Content Library that it to be deleted (replace <cl_name> with the name of the Content Library):
      
      select id,name from cl_library where name = '<cl_name>';
      
      
   3. Use the Content Library ID to find the storage ID:
      
      select * from cl_library_storage where library_id = '<ID from step 2>';
      
      
   4. Using the 2 IDs, delete the Entries for the Content Library:
      
      delete from cl_library_storage where library_id = '<ID from step 2>';

delete from cl_storage where id = '<storage_id from step 3>';

delete from cl_library where name = '<ID from step 2>';
      
      
   5. Exit psql command line using:
      
      \q
      
      

• After deleting the Content library if some Library items fails to delete and remain as stale, follow the below steps to delete them manually.

  1. select * from cl_libraryitem;   
     
     Identify the library item with the same library_id as deleted above (As in step 2 from above) and still remains in the vcdb. 
     
     Ex:
     

     26a5342c-####-####-####-28c539b3f444 | YYYY-MM-DD HH:MM:SS.106 | YYYY-MM-DD HH:MM:SS.761 | YYYY-MM-DD HH:MM:SS.507 | 19b1fa06-####-####-####-23379ec0e9c7 | cl_libraryitem_name               | ovf  | https://vc_fqdn:443/cls/vcsp/lib/3c723c9d-####-####-####-b24fb5a070c0/654b5d3c-####-####-####-f24af0dda5d7/item.json | t

     
     34513c64-####-####-####-18fc014e4d21 | YYYY-MM-DD HH:MM:SS.508 | YYYY-MM-DD HH:MM:SS.604 | YYYY-MM-DD HH:MM:SS.88  | 19b1fa06-####-####-####-23379ec0e9c7 | cl_libraryitem_name               | ovf  | https://vc_fqdn:443/cls/vcsp/lib/3c723c9d-####-####-####-b24fb5a070c0/6c4d0d95-####-####-####-34e555137cfa/item.json | t

  2. Delete the stale library items registered under stale library_id which was deleted from the above steps.
     
     delete from cl_libraryitem where id = '26a5342c-####-####-####-28c539b3f444';
     delete from cl_libraryitem where id = '34513c64-####-####-####-18fc014e4d21';

 

• Start the vpxd and content library services:
  
  service-control --start vmware-vpxd

service-control --start vmware-content-library