Unmount NFS datastore from ESXi hosts known to VCF but not part of a cluster
Article ID: 312073
Updated On:
Products
VMware Cloud Foundation
Issue/Introduction
Symptoms:
2023-02-07T10:39:09.888+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.v.m.a.RemoveNfsDatastoreAction,pool-5-thread-9] Host esxi-9.vrack.vsphere.local still has reference of datastore lcm-bundle-repo
2023-02-07T10:39:09.888+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.v.m.a.RemoveNfsDatastoreAction,pool-5-thread-9] Retry count 3 for the datastore cleanup check
2023-02-07T10:39:09.888+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.e.s.c.c.v.vsphere.VsphereClient,pool-5-thread-9] Destroying 3 open views
2023-02-07T10:39:09.918+0000 WARN [vcf_migration,0000000000000000,0000] [c.v.v.v.c.h.i.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase,pool-5-thread-9] Shutting down the connection monitor.
2023-02-07T10:39:09.919+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.v.m.a.RemoveNfsDatastoreAction,pool-5-thread-9] Unable to remove the existing lcm-bundle-repo datastore
2023-02-07T10:39:09.919+0000 WARN [vcf_migration,0000000000000000,0000] [c.v.v.v.c.h.i.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase,VLSI-client-connection-monitor-30] Interrupted, no more connection pool cleanups will be performed.
2023-02-07T10:39:09.975+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,pool-5-thread-9] [HRHVRF] FAILED_TO_DELETE_NFS_DATASTORE Failed to delete NFS Datastore lcm-bundle-repo on ESXi Host
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to delete NFS Datastore lcm-bundle-repo on ESXi Host
at com.vmware.vcf.migration.actions.RemoveNfs
DatastoreAction.execute(RemoveNfsDatastoreAction.java:127)
at com.vmware.vcf.migration.actions.Remove
NfsDatastoreAction.execute(RemoveNfsDatastoreAction.java:45)
at com.vmware.evo.sddc.orchestrator.platform.
action.FsmActionState.invoke(FsmActionState.java:62)
- NFS datastore can not mount on unmanaged commissioned ESXi hosts
Workflow fails with following exceptions observed in sddc_migration_upgrade.log
2023-02-07T10:39:09.888+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.v.m.a.RemoveNfsDatastoreAction,pool-5-thread-9] Host esxi-9.vrack.vsphere.local still has reference of datastore lcm-bundle-repo
2023-02-07T10:39:09.888+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.v.m.a.RemoveNfsDatastoreAction,pool-5-thread-9] Retry count 3 for the datastore cleanup check
2023-02-07T10:39:09.888+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.e.s.c.c.v.vsphere.VsphereClient,pool-5-thread-9] Destroying 3 open views
2023-02-07T10:39:09.918+0000 WARN [vcf_migration,0000000000000000,0000] [c.v.v.v.c.h.i.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase,pool-5-thread-9] Shutting down the connection monitor.
2023-02-07T10:39:09.919+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.v.m.a.RemoveNfsDatastoreAction,pool-5-thread-9] Unable to remove the existing lcm-bundle-repo datastore
2023-02-07T10:39:09.919+0000 WARN [vcf_migration,0000000000000000,0000] [c.v.v.v.c.h.i.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase,VLSI-client-connection-monitor-30] Interrupted, no more connection pool cleanups will be performed.
2023-02-07T10:39:09.975+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,pool-5-thread-9] [HRHVRF] FAILED_TO_DELETE_NFS_DATASTORE Failed to delete NFS Datastore lcm-bundle-repo on ESXi Host
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to delete NFS Datastore lcm-bundle-repo on ESXi Host
at com.vmware.vcf.migration.actions.RemoveNfs
DatastoreAction.execute(RemoveNfsDatastoreAction.java:127)
at com.vmware.vcf.migration.actions.Remove
NfsDatastoreAction.execute(RemoveNfsDatastoreAction.java:45)
at com.vmware.evo.sddc.orchestrator.platform.
action.FsmActionState.invoke(FsmActionState.java:62)
Environment
VMware Cloud Foundation 5.0
Vmware Cloud Foundation 4.5.1
Vmware Cloud Foundation 4.5.1
Cause
The issue can occur since the current config drift framework can not unmount NFS datastore on unmanaged hosts.
Resolution
To remove the datastore mount points from unmanaged hosts follow the below steps:
1. Use vSphere Client to login to ESXi/VC to check the mounted datastores on hosts.You can also login to ESXi host via ssh and run below command to list the mounted datastore on hosts.
2. Run below command on the ESXi hosts to remove the NFS mount point listed from step 1.
3. Run below command on ESXi hosts to verify if the NFS mount point has been removed. Also you can retry the failed workflow and verify if it goes through.
1. Use vSphere Client to login to ESXi/VC to check the mounted datastores on hosts.You can also login to ESXi host via ssh and run below command to list the mounted datastore on hosts.
2. Run below command on the ESXi hosts to remove the NFS mount point listed from step 1.
root@esxi:~] esxcli storage nfs remove --volume-name=lcm-bundle-repo
3. Run below command on ESXi hosts to verify if the NFS mount point has been removed. Also you can retry the failed workflow and verify if it goes through.
[root@esxi:~] esxcli storage nfs list
Feedback
Yes
No
Powered by
