"Permission to perform this operation was denied due to missing Privilege Datastore.FileManagement", CNS CreateVolume fails after patching VC to 8.0 U3

Products

• VMware vSphere with Tanzu

Issue/Introduction

• Newly deployed worker node fails with error "failed to create volume"
  
  
• Describing PVC shows below error message:
  
  kubectl describe pvc <pvc_name> -n <namespace>

Warning  ProvisioningFailed    58m (x14 over 79m)     csi.vsphere.vmware.com_<Supervisor_ID>_<UUID>  failed to provision volume with StorageClass "<Storage-policy-name>": rpc error: code = Internal desc = failed to create volume. Error: ServerFaultCode: Permission to perform this operation was denied.
  Normal   ExternalProvisioning  4m25s (x303 over 79m)  persistentvolume-controller                                                                   Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          55s (x29 over 79m)     csi.vsphere.vmware.com_<Supervisor_ID>_<UUID>  External provisioner is provisioning volume for claim "<namespace>/<GuestClusterName>-containerd-0"
(END) 


• CSI logs on Supervisor Cluster will show below errors - /var/log/pods/vmware-system-csi_vsphere-csi-controller-<ID>/vsphere-csi-controller/manager/xx.log

stderr F {"level":"info","time":"2024-07-18T20:25:44.706701057Z","caller":"volume/util.go:350","msg":"Extract vimfault type: +types.NoPermission. SoapFault Info: +&{{http://schemas.xmlsoap.org/soap/envelope/ Fault} ServerFaultCode Permission to perform this operation was denied. {{{{{<nil> []}}}
Datastore:datastore-<moid> Datastore.FileManagement []}}} from err +ServerFaultCode: Permission to perform this operation was denied.","TraceId":"<UUID>"}


• WCP logs on vCenter Server will show below errors - /var/log/vmware/wcp/wcpsvc.log 

error wcp [vclib/authz.go:50] [opID=66964473] Unable to set permissions [{{} <nil> [email protected] false 1031 true}] for entity Folder:group-dX. Err ServerFaultCode: The requested change cannot be completed because it could leave the system without full administrative privileges for a user or group.

Environment

vSphere with Tanzu 8.x

Cause

This issue can happen if the vpxd-extension<machine-id> is missing in ServiceProviderUsers group. 

Resolution

• To find the vCenter Server machine-id run the below command
  • /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
    
    
• Add the missing vpxd-extention user to ServiceProviderUsers group by running below command
  • /usr/lib/vmware-vmafd/bin/dir-cli group modify --name 'ServiceProviderUsers' --add "vpxd-extension-<machineid>" --login '<SSO_ADMIN_USER>'

Note:

• • • Replace the "<machineid>" with the output from the previous command
    • Replace "<SSO_ADMIN_USER>" - [email protected] or administrator@<custom-sso-domain-name>

Additional Information

vCenter Server 8.0U3