VMFS datastores cannot have their capacity reduced without destroying the original datastore.
This is because data is not always sequentially written, in other words, the blocks used are not sequential, so reducing the available space for the VMFS datastore could leave used blocks outside the new size, which would cause data loss and filesystem corruption.