Error: "Internal Server Error" when Refreshing vCenter

Products

• VMware Cloud Director

Issue/Introduction

• Tasks initiated from VMware Cloud Director are taking a long time to complete. 
• Tasks initiated from VMware Cloud Director are taking a long time to reach the vCenter Server.
• Attempting to Refresh VirtualCenter from VMware Cloud Director Provider UI results in the following error...
  
  "Internal Server Error - Expected completed future, but received future which is still In progress
com.vmware.vcloud.api.presentation.service.lnternalServerErrorException: Internal Server Error"

Environment

• 10.x
• 10.3.3

Cause

The activity scheduler built into VMware Cloud Director is not operating properly. 

Resolution

Use the cell-management-tool fail-tasks command to clear all scheduled jobs, ensuring they are re-scheduled appropriately.


Note: Start the following steps with the Standby Cells

1. Obtain the cell pid: For use with the cell-management-tool commands:
   
   cat /var/run/vmware-vcd-cell.pid
   
   
2. Queisce Cell: This ensure the cell does not take anymore jobs.
   
   /opt/vmware/vcloud-director/bin/cell-management-tool cell -i <cell pid> -q true
   
   Example:
   /opt/vmware/vcloud-director/bin/cell-management-tool cell -i 111 -q true
   
   
3. Verify the job count is 0:
   
   /opt/vmware/vcloud-director/bin/cell-management-tool cell -i <cell pid> -t
   
   
4. Maintenance Mode:
   
   /opt/vmware/vcloud-director/bin/cell-management-tool cell -i <cell pid> -m true
   
   
5. Shutdown Cell:
   
   /opt/vmware/vcloud-director/bin/cell-management-tool cell -i <cell pid> -s
   
   
6. Repeat Steps 1 - 5 on the Primary Cell. 
   
   
7. Run the Fail Tasks command on the Primary Cell Only:
   
   /opt/vmware/vcloud-director/bin/cell-management-tool cell -i <cell pid> fail-tasks -m "admin shutdown"
   
   
8. Reboot the cells starting with the Primary then Standby(s).
   
   
9. Verify the jobs have been rescheduled using the cell-management-tool
   
   
   1. See Step 1 to gather the new cell pid
      
      
   2. List Active Scheduled Jobs:
      
      /opt/vmware/vcloud-director/bin/cell-management-tool cell - i <cell pid> -tt