CNS/Tanzu VMs are not able to fully deploy. VC "Recent Tasks" displays ongoing "Reconfigure vSAN file share. Cannot complete the operation" messages.
search cancel

CNS/Tanzu VMs are not able to fully deploy. VC "Recent Tasks" displays ongoing "Reconfigure vSAN file share. Cannot complete the operation" messages.

book

Article ID: 405546

calendar_today

Updated On:

Products

VMware vSAN 8.x VMware vCenter Server 8.0

Issue/Introduction

- During Tanzu upgrade "Reconfiguring vSAN File Service" task may repeatedly fail with the following error message: 

A specified parameter was not correct: config.labels Failed to reconfigure file share: Total file share label count is larger than 1000.

- Additionally, vSAN health service log (/var/log/vmware/vsan-health/vmware-vsan-health-service.log on vCenter Appliance) may log lines similar to:

INFO vsan-mgmt[08924] [VsanScheduler::_ThreadMain opID=xxxxxxxxx] Executing itemListHead: RecfgOrRemoveFileShare-file:xxxxx-xxxxxx-xxxx-xxx: func: _ReconfigureFileShare, {'cluster': 'vim.Clust
erComputeResource:domain-xxxxxxx', 'shareUuid': 'file:xxxxxx-xxxx-xxxxxxx-xxxxx', 'config': (vim.vsan.FileShareConfig) {
 
.....
 
}, 'deleteLabelKeys': (str) [
  'cns.k8s.pv[2].referredEntity.namespace',
  'cns.k8s.pv[2].referredEntity.clusterId',
  'cns.k8s.pv[2].referredEntity.pvc.name'
], 'force': True, 'vsanTask': <VsanHealthHelpers.VsanTask object at 0x7fb3794c4a90>, 'taskKey': 'com.vmware.vsan.fileservice.tasks.reconfigurefileshare'}, {}
 

Environment

VMware vSAN 8.0.x

VMware vCenter Server 8.0.x

Resolution

To workaround the issue a cleanup of share labels should be performed:
 
1. Obtain share UUID from share detail UI page, e.g.: file:xxxxx-xxxx-xxxxx-xxxx
2. Log into one of the ESX host
3. Obtain (RootFS) UUID running command:
 
cmmds-tool find -t VDFS ( The rootfs volume has content value (l1), see an example below.)
 
e.g
-- owner=xxxx-xxxxx-xxxx-xxxxx(Health: Healthy) uuid=xxxx-xxxxx-xxxx-xxxxxx type=VDFS rev=100 minHostVer=0  [content = (l1)], errorStr=(null) <<<<<<<<<<<< ROOT 
 
4. Identify the ESXI host owner running the command:
 
cmmds-tool find -f json -t HOSTNAME |grep -E "uuid|content"|sed 'N;s/\n/ /'|awk -F \" '{print $10": " $4}'|sort
 
 
5. On ESXI host owner run the command to access to the share labels:
 
cd /vmfs/volumes/vdfsDatastore/<rootfsUuid>/volumes/default/<fsDomainUuid>/share_labels/<shareUuid>/
 
6. Verify if there are more than 1000 files inside the folder, each file represents a share label
 
run: ls <share-uuid> | wc -l 
 
7. Remove files with command: 
 
rm -rf <files> 
 
 
Please note: This operation will remove labels only and not the actual data.