Failing to add new disk to deployment - Request is never leaving VRA or reaching vCenter

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Adding a disk to an existing deployment fails
  
  
• You see the CREATE task in the Request history, then two minutes later the task fails with: "Error=java.lang.IllegalStateException: jakarta.xml.ws.WebServiceException: java.net.SocketTimeoutException: Read timed out"
  
  
• You do not see the associated task starting in vCenter
  
  
• In the provisioning-service-app logs (/services-logs/prelude/provisioning-service-app/files-logs/provisioning-service-app.log), you may see an error similar to:
  "Exception while processing create callback response for resource New_Disk_Name_#[0] and deployment ########-####-####-####-############. Error=java.lang.IllegalStateException: jakarta.xml.ws.WebServiceException: java.net.SocketTimeoutException: Read timed out"

Environment

Aria Automation 8.x

Resolution

Restart the provisioning-service-app pods by scaling them down and back up. 

1. Run the following command to scale down and back up, the provisioning-service pod(s) for all nodes. 
       Note: This only needs to be run once from one node in the cluster
   
   kubectl scale deployment provisioning-service-app -n prelude --replicas=0 && kubectl scale deployment provisioning-service-app -n prelude --replicas=$(kubectl get nodes --no-headers | wc -l)
   
   
2. You can run the following command to monitor the status of provisioning-service-app pod(s) until all provisioning-service-app pod(s) are up and running:
   
   kubectl -n prelude get pods | grep provisioning-service-app
   
   
3. Check the pods to ensure that all of the pods are running again
   
   kubectl -n prelude get pods -o wide |grep provisioning-service-app
   
   
4. After all of the provisioning-service-app pods are running, run the Day 2 action to add a disk to the deployment that was failing.