CNS/Tanzu VMs are not able to fully deploy. VC "Recent Tasks" displays ongoing "Reconfigure vSAN file share. Cannot complete the operation" messages.

Products

• VMware vSAN 8.x
• VMware vCenter Server 8.0

Issue/Introduction

- During Tanzu upgrade "Reconfiguring vSAN File Service" task may repeatedly fail with the following error message: 

A specified parameter was not correct: config.labels Failed to reconfigure file share: Total file share label count is larger than 1000.

- Additionally, vSAN health service log (/var/log/vmware/vmware-vsan-health-service.log on vCenter Appliance) may log lines similar to:

INFO vsan-mgmt[08924] [VsanScheduler::_ThreadMain opID=xxxxxxxxx] Executing itemListHead: RecfgOrRemoveFileShare-file:xxxxx-xxxxxx-xxxx-xxx: func: _ReconfigureFileShare, {'cluster': 'vim.Clust
erComputeResource:domain-xxxxxxx', 'shareUuid': 'file:xxxxxx-xxxx-xxxxxxx-xxxxx', 'config': (vim.vsan.FileShareConfig) {

 

.....

 

}, 'deleteLabelKeys': (str) [
  'cns.k8s.pv[2].referredEntity.namespace',
  'cns.k8s.pv[2].referredEntity.clusterId',
  'cns.k8s.pv[2].referredEntity.pvc.name'
], 'force': True, 'vsanTask': <VsanHealthHelpers.VsanTask object at 0x7fb3794c4a90>, 'taskKey': 'com.vmware.vsan.fileservice.tasks.reconfigurefileshare'}, {}

 

Environment

VMware vSAN 8.0.x

VMware vCenter Server 8.0.x

Resolution

To workaround the issue a cleanup of share labels should be performed:

 

1. Obtain share UUID from share detail UI page, e.g.: file:xxxxx-xxxx-xxxxx-xxxx

2. Log into one of the ESX host

3. Obtain (RootFS) UUID running command:

 

cmmds-tool find -t VDFS ( The rootfs volume has content value (l1), see an example below.)

 

e.g

-- owner=xxxx-xxxxx-xxxx-xxxxx(Health: Healthy) uuid=xxxx-xxxxx-xxxx-xxxxxx type=VDFS rev=100 minHostVer=0  [content = (l1)], errorStr=(null) <<<<<<<<<<<< ROOT 

 

4. Identify the ESXI host owner running the command:

 

cmmds-tool find -f json -t HOSTNAME |grep -E "uuid|content"|sed 'N;s/\n/ /'|awk -F \" '{print $10": " $4}'|sort

 

 

5. On ESXI host owner run the command to access to the share labels:

 

cd /vmfs/volumes/vdfsDatastore/<rootfsUuid>/volumes/default/<fsDomainUuid>/share_labels/<shareUuid>/

 

6. Verify if there are more than 1000 files inside the folder, each file represents a share label

 

run: ls <share-uuid> | wc -l 

 

7. Remove files with command: 

 

rm -rf <files> 

 

 

Please note: This operation will remove labels only and not the actual data.