Migrated in-tree vSphere volumes do not have the FCD control flag - keepAfterDeleteVm. So if a worker node gets deleted, then all migrated in-tree volumes being attached to the node will be removed automatically as well.
There are two prerequisites:
Users are supposed to disable BOSH resurrection before performing any operation.
$ bosh update-resurrection off
If any worker node is unresponsive during the operation, then the operation (i.e. cluster-upgrade) may fail. Please follow Step 2 to workaround it when running into such a situation, and retry the previously failed operation afterward.
If there isn't any unresponsive worker node, then no action is required in this step.
If you want to test the steps in your test environment, then you can simulate them by executing the below command on any worker node.
# sv stop agent
Follow the steps below to fix the unresponsive node.
$ export NODE_NAME=32c1090d-cd3c-4d24-93d4-3fdf0153dcff
$ kubectl get pod -A --field-selector spec.nodeName=${NODE_NAME} -o=json | jq -c '.items[] | {name: .metadata.name, namespace: .metadata.namespace, claimName: .spec | select( has ("volumes") ).volumes[] | select( has ("persistentVolumeClaim") ).persistentVolumeClaim.claimName }'
Please replace the node name(32c1090d-cd3c-4d24-93d4-3fdf0153dcff ) in the example above with your unresponsive node.
The output is something like below,
{"name":"nginx-deployment-7d7948fb9c-djhl4","namespace":"default","claimName":"pvcsc-vsan"}
3. Forcibly delete all PODs returned at the above step
$ kubectl delete pod nginx-deployment-7d7948fb9c-djhl4 --force
4. Wait for all PODs deleted above to be running again.
If users run into the known node non-gracefully shutdown issue, then please follow https://kb.vmware.com/s/article/85213 to work around it.
5. Recreate the unresponsive worker node. (Use your service instance id & worker node id)$ bosh -d service-instance_7197b487-239d-
4305-a63e-432b06376f29 recreate worker/c00197e7-5a29-4b14-98c6-cc10d3209524 --fix
When all clusters are successfully upgraded to a safe version, enable bosh resurrection again.
$ bosh update-resurrection on