CNS tasks in vCenter Server frequently fail with the error, A specified parameter is not correct.

Products

• VMware vCenter Server 8.0

Issue/Introduction

After upgrading to vCenter 8.x, you see frequent tasks related to Cloud Native Storage (CNS) fail with the error,

A specified parameter is not correct.

/var/log/vmware/vpxd/vpxd.log reports the below error

YYYY-MM-DDT23:36:51.974Z error vpxd[304593] [Originator@6876 sub=Default opID=5bed11f5] [VpxLRO] -- ERROR lro-73792948 -- 52ed579a-dbd8-3152-b240-#####(522b9d6b-3679-b555-e7b2-#####) -- VStorageObjectManager --vim.vslm.vcenter.VStorageObjectManager.retrieveVStorageObject: :vmodl.fault.InvalidArgument
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    invalidProperty = ""
-->    msg = "Received SOAP response fault from [<<cs p:00007ff75402e1e0, PIPE:/var/run/envoy-hgw/hgw-pipe>, /hgw/host-23/vpxa>]: retrieveVStorageObject
--> Received SOAP response fault from [<SSL(<io_obj t:N7Vmacore6System19TCPSocketObjectAsioE, h:18, <TCP '127.0.0.1 : 50031'>, <TCP '127.0.0.1 : 443'>>), /sdk>]: retrieveVStorageObject
--> A specified parameter was not correct: "
--> }
--> Args:
-->
--> Arg id:
--> (vim.vslm.ID) {
-->    id = "file:fe4f1693-0dfd-42f4-a39f-####"
--> }
--> Arg datastore:
--> 'vim.Datastore:datastore-73'
--> Arg diskInfoFlags:
--> (string) []

Cause

After upgrading to vCenter 8.x, CNS runs a full sync on datastores to clean up stale block volumes.

When a datastore has both file and block volumes, the CNS periodic sync function may incorrectly indicate file volumes as stale block volumes.

In such cases, CNS calls block volume API retrieveVStorageObject on the file volumes, which fails with "A specified parameter was not correct" error.
CNS adds the file volumes to the CNS retry table vpx_storage_volume_update and continues retrying the retrieveVStorageObject call.
This causes the frequent "A specified parameter was not correct" messages in vCenter Server

Resolution

To resolve the issue,

1. Ensure you have a working backup of the vCenter Server.
2. Connect to VCDB in vCenter

   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

    

3. Execute the below SQL query.
   

   select * from cns.vpx_storage_volume_update ;

4. Delete rows whose volume_id start with the pattern ""file:"" from the vns.vpx_storage_volume_update table.
   

   delete from cns.vpx_storage_volume_update where volume_id like 'file:%';

5. After the file ids are cleared, the errors should not be seen in vCenter Server.
6. The workaround might have to be applied every 5-6 hours when fullSync happens.

This Issue has been fixed in vCenter Server 8.0 update 3e