The “Decommission vCenter Server” task failed upon retry.

Products

• VMware SDDC Manager
• VMware vCenter Server

Issue/Introduction

Symptoms

• Due to an issue, the “Decommission vCenter Server” task was re-executed from SDDC Manager.
  
  
• The following error is recorded to the SDDC Manager domainmanager.log :

  YYYY-MM-DDTHH:MM:SS.000+0000 ERROR [vcf_dm,aaaaaaaaa] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-5]  [TU1P8P] SSO_CANNOT_DECOMMISSION_NODE Node <　vCenter FQDN > decommission could not be initiated
  com.vmware.evo.sddc.common.core.error.InvalidStateException: Node < vCenter FQDN > decommission could not be initiated
   <snip>
  Caused by: org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: "{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.not_allowed_in_current_state","decommission$task"],"default_message":"Invalid error com.vmware.vapi.std.errors.not_allowed_in_current_state reported from method decommission$task.","id":"vapi.method.status.errors.invalid"},{"args":[],"default_message":"Duplicate decommission call. Ignoring call, as decommission is already in progress.","id":"com.vmware.appliance.DuplicateCallCheck"}]}}"

  Note : If other errors such as “Fetched topology with 3 nodes, contains nodes with invalid partner count, invalid ones” are recorded in domainmanager.log, resolving this KB's resolution alone will not fix the issue unless those problems are addressed.
  
  
• The file /var/log/vmware/decommission/decommission_status.json exists on the vCenter < vCenter FQDN > mentioned in the above error.

Environment

VMware SDDC Manager 4.5
VMware SDDC Manager 5.1
VMware SDDC Manager 5.2
VMware vCenter Server 7.0
VMware vCenter Server 8.0

 

Cause

This is because leaving decommission_status.json behind causes the decommissioning task to be identified as duplicated.

Resolution

The background for retry “Decommission vCenter Server” is that other issues have occurred, and since those issues have been resolved, it is assumed this task will be retry.
Please verify that existing the snapshot of the vCenter and SDDC Manager taken before fixing the other issues before implementing the following workaround.

1. Back up the files under /var/log/vmware/decommission/ in vCenter as listed in domainmanager.log.
   
   
2. SSH into vCenter as the root user and delete the files using the following command:
   

   rm -rf /var/log/vmware/decommission/*

3. Restart the applmgmt service on vCenter.
   

   service-control --restart applmgmt

4. Re-execute the task from SDDC Manager.

Additional Information

• “Decommission vCenter Server” タスクが再試行時に失敗する