Failed to delete the content library item via vSphere web client

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

Can't delete or update content library item in vSphere webclient, below errors found in vCenter vpxd.log:

Log directory is var/log/vmware/vpxd/vpxd.log.

info vpxd[7FCC2D195700] [Originator@6876 sub=Default opID=DatastoreExplorerMediator-removeOnMultiEntity-59878-ngc:70005236-e8] [VpxLRO] -- ERROR task-22259 -- FileManager -- vim.FileManager.delete: vim.fault.CannotDeleteFile:
--> Result:
--> (vim.fault.CannotDeleteFile) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    file = "[DATASTORE] contentlib-<UUID>/<UUID>/vm_name.vmdk"
-->    msg = "Received SOAP response fault from [<cs p:00007fcbb444f130, TCP:Server:443>]: Delete
--> "
--> }
--> Args:
-->
--> Arg datacenter:
--> 'vim.Datacenter:<UUID>:datacenter-21'
--> Arg datastorePath:
--> "[VOL01_VIRTUAL_01] contentlib-<UUID>/<UUID>/vm_name.vmdk"
--> Arg fileType:
--> "VirtualDisk"

You also may see errors similar to the below in the vpxa.log file:

info vpxa[9867B70] [Originator@6876 sub=DiskLib opID=dam-auto-generated: DatastoreFilesViewMediator:dr-663:FilePropertyProvider:201242:8337-59866-ngc:70005235-58-2b] DISKLIB-LINK  : "/vmfs/volumes/<UUID>/contentlib-<UUID>/<UUID>/vm_name-1_<UUID>-flat.vmdk" : failed to open (The file specified is not a virtual disk).

info vpxa[8C39B70] [Originator@6876 sub=Libs opID=DatastoreExplorerMediator-removeOnMultiEntity-59878-ngc:70005236-e8-e5] OBJLIB-FILEBE :FileBEUnlink : Failed to unlink the file '/vmfs/volumes/<UUID>/contentlib-<UUID>/<UUID>/vm_name-1_<UUID>-flat.vmdk' : 1048580

info vpxa[8C39B70] [Originator@6876 sub=DiskLib opID=DatastoreExplorerMediator-removeOnMultiEntity-59878-ngc:70005236-e8-e5] DISKLIB-LIB   : Cannot remove extent `/vmfs/volumes/<UUID>/contentlib-<UUID>/<UUID>/vm_name-1_<UUID>-flat.vmdk': Device or resource busy

error vpxa[8C39B70] [Originator@6876 sub=vpxaMoService opID=DatastoreExplorerMediator-removeOnMultiEntity-59878-ngc:70005236-e8-e5] [DeleteFile] Received exception: N7Vmacore31FileIOPermissionDeniedExceptionE(FileIO error: Permission denied for file  : /vmfs/volumes/<UUID>/contentlib-<UUID>/<UUID>/vm_name-1_<UUID>.vmdk)

verbose vpxa[8D41B70] [Originator@6876 sub=vpxaMoService opID=dam-auto-generated: DatastoreFilesViewMediator:dr-673:FilePropertyProvider:201242:8351-59887-ngc:70005239-a4-b6] SearchDatastore: ds:///vmfs/volumes/<UUID>/contentlib-<UUID>/<UUID> false [N3Vim4Host16DatastoreBrowser10SearchSpecE:0x04061c50]

info vpxa[8D41B70] [Originator@6876 sub=DiskLib opID=dam-auto-generated: DatastoreFilesViewMediator:dr-673:FilePropertyProvider:201242:8351-59887-ngc:70005239-a4-b6] DISKLIB-LINK  : "/vmfs/volumes/<UUID>/contentlib-<UUID>/<UUID>/vm_name-1_<UUID>-flat.vmdk" : failed to open (The file specified is not a virtual disk).

Environment

VMware vCenter Server Appliance 6.7 / 7.x / 8.x

Cause

Possible Database corruption on content library record, or files were incorrectly removed outside of the UI.

Resolution

Ensure you take a VCSA database backup and VCSA virtual machine snapshot before proceed.

If an entire library needs to be removed, perform the following:

1. SSH to the VCSA and login to the embedded Postgres DB:

su - postgres /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

2. Perform the below query to get the whole picture of content library:

select * from cl_storage;
select * from cl_library；
select * from cl_library_storage;
select * from cl_libraryitem;
select * from cl_libraryitemversion;
select * from cl_libraryitemversionfile_view; 
select * from cl_librarywithitems_view;

3. If any library item should be exported in advance, Please refer to Export an Item from a Content Library to Your Local Computer 

4. If no library item should be left, execute below SQL statements to remove all items from the library:

delete from cl_libraryitemversion where libraryitemid in (select id from cl_libraryitem where libraryid ='library id');
delete from cl_libraryitem where libraryid ='library id';

5. Execute the below SQL statements to delete target content library:

delete from cl_library_storage where library_id ='library id';
delete from cl_library where name='library name';

6. Confirm that content library is cleaned up by using below commands:

select * from cl_library_storage where library_id = (select id from cl_library where name='library name'); 
select * from cl_libraryitemversionfile_view; 
select * from cl_librarywithitems_view;

8. Open vSphere web client and check if the target library removed.

 

If only one or more specific Content Library Items needs to be deleted, use the following commands:

1. SSH to the VCSA and login to the embedded Postgres DB:

su - postgres /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

2. Get a list of the library items to obtain the proper ID for the item you wish to delete:

select * from cl_libraryitem;

3. Remove the item(s) (in this example the ID is '00aa11bb-22cc-33dd-44ee-55ff66gg77hh'; replace this with the actual ID that needs to be removed):

delete from cl_libraryitem where id='00aa11bb-22cc-33dd-44ee-55ff66gg77hh';