CNS storage plugin error: "A specified parameter was not correct"

Products

• VMware Telco Cloud Platform Advanced
• VMware Telco Cloud Platform Essentials

Issue/Introduction

In vCenter's tasks, you frequently receive messages from com.vmware.cns indicating 'A specified parameter was not correct.'

You do not have any impact to the running workloads.

New workloads can be deployed without issue.

In the vpxd.log on vCenter you see log messages like the one below:

2024-07-25T20:14:04.137Z error vpxd[3920167] [Originator@6876 sub=Default opID=5ecfbe90] [VpxLRO] -- ERROR lro-55701 -- 52293931-aaaa-bbbb-cccc-fd5f66c25994(52556658-aaaa-bbbb-cccc-e292df06544d) -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.retrieveVStorageObject: :vmodl.fault.InvalidArgument
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    invalidProperty = ""
-->    msg = "Received SOAP response fault from [<<io_obj p:0x0000000123456789, h:64, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-12345/vpxa>]: retrieveVStorageObject
--> Received SOAP response fault from [<<io_obj p:0x0000000123456789, h:18, <TCP '127.0.0.1 : 51572'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: retrieveVStorageObject
--> A specified parameter was not correct: "
--> }
--> Args:
-->
--> Arg id:
--> (vim.vslm.ID) {
-->    id = "file:abcd1234-aaaa-bbbb-cccc-123456789abc"
--> }
--> Arg datastore:
--> 'vim.Datastore:datastore-01'
--> Arg diskInfoFlags:
--> (string) []

Environment

vCenter/ESXi 8.0U3 and earlier

Cause

This is due to a bug in CNS.

CNS maintains an in-memory volume cache which should only contain block volumes. When vsan-health is restarted, both file and block volumes from the CNS database get loaded into the in-memory CNS volume cache, however only block volumes should be loaded. During a full-sync the CNS tries to clean the stale block volumes present in its volume cache. After the restart, file volumes present in the volume cache are incorrectly treated as stale block volumes and clean up is attempted against these volumes. The clean up fails because block volume APIs are invoked against the file volumes and this appears in the vCenter tasks as the message 'a specified parameter was incorrect.'

Resolution

This will be fixed in a future release of CNS.

As a workaround, you can manually clean up the VCDB following the steps below:

1) Connect to the VCDB:
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

2) Run the following SQL query:
select * from cns.vpx_storage_volume_update ;

3) If there are any rows where the "volume_id" column begins with "file:", delete those rows.
delete from cns.vpx_storage_volume_update where volume_id like 'file:%';

Note: The table vpx_storage_volume_update is meant to be used by only non-file file IDs, so there's no danger in deleting those rows with file ids.

Once the table is cleared of file IDs, we should no longer see these failures. However, if vsan-health is restarted, these entries may re-populate, requiring reapplication of this workaround.