Error message "Exceeded the number of devices that can be detached. Clean up all stale detach entries" appears when detaching a LUN from ESXi
Article ID: 324321
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
- When you attempt to detach a LUN from the vSphere Web Client or the ESXi Host Client you receive an error message: "Cannot change the host configuration"
- Executing the command esxcli storage core device detached list returns no results
- In the vmkernel.log you could see entries similar to:
Environment
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.5
Cause
This message could appear when multiple LUNs flagged as "perennially reserved" have been detached in the past without firstly remove the perennially reserved flag. The ESXi currently manages a list in the VSI nodes to keep track of these LUNs and when the LUN is detached this list is not automatically updated to remove the device from it. The list can have up to 512 devices, when you attempt to detach a 513th device the error appears.
Resolution
VMware is currently working on improving this behavior.
If you are currently experiencing the problem reported in this KB then remove the perennially reserved flag of LUNs that were removed in the past.
Run this command to find the naa.id devices listed in the VSI nodes
# vsish -e ls /storage/scsifw/persistDeviceAttrs/uids
Run this command to verify if there are 512 devices listed already:
# vsish -e ls /storage/scsifw/persistDeviceAttrs/uids | wc -l
Please note: that devices listed under uids, may have multiple attributes, including perennially reservation.
The listing of an naa ID does not automatically indicate that the LUN is perennially reserved.
/storage/scsifw/persistDeviceAttrs/uids/> get naa.624a93704a136e62ff54907c002baf51
Device Persistent Attributes {
detached:0
perennially reserved:0
not shared:1
In this example the LUN is not reserved, yet the naa ID does appear under uids, because one of the 3 attributes are TRUE.
Identify the naa.ids that where detached in the past and do not exist in the environment anymore.
Run this command to remove the perennially reserved flag on those naa.id:
# esxcli storage core device setconfig -d naa.id --perennially-reserved=false
Now you could attempt to detach the desired LUN again.
To avoid this problem from happening in the future, please follow the steps of KB Article 2004605 to detach the LUN correctly from an ESXi and as per the the Related Information of that KB the flag of "perennially reserved" needs to be removed to prevent a successful unmount/detach of the LUN.
Workaround:
You could reboot the ESXi to clean some entries that are not persistent from the VSI nodes and then be able to detach the LUN. However, those entries that are persistent will have to be manually removed with the process in the Resolution section. It is highly recommended that this clean up is performed in the environment to avoid this problem from happening in the future.
If you are currently experiencing the problem reported in this KB then remove the perennially reserved flag of LUNs that were removed in the past.
Run this command to find the naa.id devices listed in the VSI nodes
# vsish -e ls /storage/scsifw/persistDeviceAttrs/uids
Run this command to verify if there are 512 devices listed already:
# vsish -e ls /storage/scsifw/persistDeviceAttrs/uids | wc -l
Please note: that devices listed under uids, may have multiple attributes, including perennially reservation.
The listing of an naa ID does not automatically indicate that the LUN is perennially reserved.
/storage/scsifw/persistDeviceAttrs/uids/> get naa.624a93704a136e62ff54907c002baf51
Device Persistent Attributes {
detached:0
perennially reserved:0
not shared:1
In this example the LUN is not reserved, yet the naa ID does appear under uids, because one of the 3 attributes are TRUE.
Identify the naa.ids that where detached in the past and do not exist in the environment anymore.
Run this command to remove the perennially reserved flag on those naa.id:
# esxcli storage core device setconfig -d naa.id --perennially-reserved=false
Now you could attempt to detach the desired LUN again.
To avoid this problem from happening in the future, please follow the steps of KB Article 2004605 to detach the LUN correctly from an ESXi and as per the the Related Information of that KB the flag of "perennially reserved" needs to be removed to prevent a successful unmount/detach of the LUN.
Workaround:
You could reboot the ESXi to clean some entries that are not persistent from the VSI nodes and then be able to detach the LUN. However, those entries that are persistent will have to be manually removed with the process in the Resolution section. It is highly recommended that this clean up is performed in the environment to avoid this problem from happening in the future.
Additional Information
Impact/Risks:
Make sure that the perennially reserved flag is only removed from LUNs that do not exist in the environment anymore. If you remove the flag of a LUN that is currently used as RDM, the next time the ESXi is booted it will experience long time to start as per this KB 1016106
Make sure that the perennially reserved flag is only removed from LUNs that do not exist in the environment anymore. If you remove the flag of a LUN that is currently used as RDM, the next time the ESXi is booted it will experience long time to start as per this KB 1016106
Feedback
Yes
No
Powered by
