How to remove a failed workflow in VMware Cloud Foundation

Products

• VMware Cloud Foundation

Issue/Introduction

This article provides steps to clean up a failed workflow in VMware Cloud Foundation

Note: In VMware Cloud Foundation 2.x there is a failed Workflow state in the SDDC Manager UI > Status > Workflows page. In VMware Cloud Foundation 3.x and 4.x there is a failed Workflow state in the Tasks list.

Environment

VMware Cloud Foundation 3.8.x
VMware Cloud Foundation 2.x
VMware Cloud Foundation 4.x
VMware Cloud Foundation 3.9.x
VMware Cloud Foundation 3.10.x

Resolution

This is a known issue affecting VMware Cloud Foundation. Currently, there is no resolution.

Workaround:

To workaround this issue in VMware Cloud Foundation 2.x:

1. Download ZooViewer from https://github.com/HelloKittyNII/ZooViewer/blob/master/ZooViewer.zip
2. Disable the firewall by issuing this command: 

systemctl stop ​iptables 

3. Find the failed workflow ID in the SDDC Manager UI.
   1. Click on the failed task's link after expanding the workflow on the status page. 
   2. Find the workflow ID in the browser URL.  
4. Extract the contents of the zooviewer.zip file.
5. Execute the zooviewer.bat file (windows, from command prompt as administrator or zooviewer.sh for Linux) from the extracted files.
6. In the Enter the connection string window, enter the IP address

192.168.100.40:2181

7. Browse to /Workloads/Workflows in Zooviewer and select the workflow in question from step 3. 
8. Verify if it is the correct workflow and click Delete Node.
9. Enable the firewall by issuing this command: 

systemctl start iptables

Note: It may take several minutes before the ZooKeeper data is fully populated. 
 

To workaround this issue in VMware Cloud Foundation 3.x and 4.x:

1. Find the failed workflow ID in the SDDC Manager UI.
   1. Expand the failed Workflow in the Tasks list and then click on the View Subtasks link. 
   2. Find the workflow ID in the browser URL. There will be something similar to: (monitoring-panel:monitoring/tasks/subtasks/c6db71f0-c64a-11e8-a84b-93b7cda0e148). c6db71f0-c64a-11e8-a84b-93b7cda0e148 is the Workflow ID.
2. Log in to the SDDC Manager VM as the vcf user and then issue su - to switch to the root user.
3. Issue a command similar to the following to delete the failed Workflow:

curl -X DELETE http://localhost/tasks/registrations/c6db71f0-c64a-11e8-a84b-93b7cda0e148
 

Note: Replace c6db71f0-c64a-11e8-a84b-93b7cda0e148 with the Workflow ID value obtained in Step 1b.

Additional Information

Steps to delete a failed workflow in VMware Cloud Foundation 2.x from Zookeeper Using command line:

1. Identify the failed workflow task ID.
   • login to the SDDC Manager Controller UI: https://ipaddress:8443/vrm-ui
   • Click on Status >> Workflow Tasks >> Failed Tasks.
   • Click on the failed task >> The browser URL will show the failed Workflow ID.
2. SSH to the SDDC Manager Controller VM.
3. Run zkCli.sh to connect to zookeeper.
4. List the Workflows by issuing a command similar to the following:

ls /Workloads/Workflows/ed535779-9029-4186-9362-35535bda5373/--parameters

5. Run a command similar to the following to remove the failed workflow:

rmr /Workloads/Workflows/ed535779-9029-4186-9362-35535bda5373

Note: Validate that the Workflow was deleted by re-issuing the command from Step 4. There should be an output stating "Node does not exist".

Impact/Risks:
Choosing the wrong workflow will lead to serious issues. Ensure the correct workflow ID is chosen and always have a backup/snapshot of the SDDC Manager Controller virtual machine.