This issue may occur when the hosts that can access the shared VMFS3 datastore that you are upgrading include hosts running ESX/ESXi version 4.x or earlier.Operation failed, diagnostics report: Unable to Upgrade Filesystem: Device or resource busy
Because legacy hosts do not support VMFS5, to avoid this issue, make sure that all hosts accessing the datastore are upgraded to ESXi version 5.0.
Note: This issue may also happen to any other host that is not ESXi 5.0 and has the same volume mounted either in the same cluster or in a different cluster, managed by same/different vCenter server, which was ignored/overlooked during the troubleshooting. In certain situations, you can remove the volume from that ESXi host or upgrade the volume to resolve the issue.