"A general system error occurred: Image is not valid. Software Solution com.vmware.vsphere-wcp with version 9.0.1.##.#.#-######## cannot be found in depot", Image authoring in vLCM may fail even when the corresponding depots are imported into vLCM

Products

• VMware vCenter Server
• VMware Cloud Foundation
• VMware vSphere Kubernetes Service

Issue/Introduction

• Adding Supervisor in 9.x fails with error "A general system error occurred: Image is not valid. Software Solution com.vmware.vsphere-wcp with version 9.0.1.##.#.#-######## cannot be found in depot".
  
  
  
  
• Same error can be observed while initiating any workflow which involves image authoring in vLCM. The task might fails with error "A general system error occurred: Image is not valid. Software Solution <Component Name> with version <Version> cannot be found in depot".
• vLCM logs will show entries similar to below snippets :
  
  /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

YYYY-MM-DDTHH:MM:SS.###Z info vmware-vum-server[#######] [Originator@6876 sub=DraftsManager] [DraftsManager 2713] Draft validation results: {
-->     "errors": [
-->         {
-->             "id": "com.vmware.vcIntegrity.lifecycle.EsxImage.SolutionNotFound",
-->             "message": {
-->                 "args": [
-->                     "com.vmware.vsphere-wcp",
-->                     "9.0.1.##.#.#-########",
-->                     "",
-->                     ""
-->                 ],
-->                 "default_message": "Software Solution com.vmware.vsphere-wcp with version 9.0.1.##.#.#-######## cannot be found in depot.",
-->                 "id": "com.vmware.vcIntegrity.lifecycle.EsxImage.SolutionNotFound",
-->                 "localized": null,
-->                 "params": null
-->             },
-->             "originator": null,
-->             "resolution": null,
-->             "retriable": null,
-->             "time": "YYYY-MM-DDTHH:MM:SS.###Z",
-->             "type": null
-->         }
-->     ],
  
  
• This issue is more likely to be seen in VCF environments where depots synced into the vLCM can come from multiple sources like SDDC-M and the Broadcom depot.

Environment

VCF 9.x

Cause

• In rare cases when multiple offline and online depots are imported into vLCM, the metadata of one depot can overwrite the metadata of another depot which was imported earlier. This effectively removes the earlier depot and trying to set a desired state using the depot which was deleted will run into an error saying that such a depot cannot be found during the draft validation.
• vLCM logs will show content already exists in the DB with a UUID. 

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

YYYY-MM-DDTHH:MM:SS.###Z error vmware-vum-server[#######] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.CreateOfflineDepotTask] [CreateOfflineDepotTask 779] Offline depot content already exists in the DB with ID : ########################################.

Resolution

Broadcom Engineering is aware about this issue and working towards fixing this issue in a future release.

Workaround

This workaround should be applied only after confirming that an overwrite has happened in the vLCM depot which can be verified in the logs. Same error "Depot not found" can happen in situation where the said depot is not imported into the vLCM as well.

1. Login to vCenter Server using SSH client.
2. Verify from the vLCM logs on the VC that said depot was actually imported into vLCM. In the logs, if the offline depot that is being imported already exists, one of the signature lines that you will see is as shown below. 
   
   /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

YYYY-MM-DDTHH:MM:SS.###Z error vmware-vum-server[#######] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.CreateOfflineDepotTask] [CreateOfflineDepotTask 779] Offline depot content already exists in the DB with ID : B0B9326BD84BA41BCE40538B89669E243D6C707E
   
   Note: There can be multiple such lines in the logs which *may* indicate overwrite of multiple depots. Note down all the depot ID shown in the log. In this case it is "B0B9326BD84BA41BCE40538B89669E243D6C707E". This depot ID will differ for each depot.
   
   
3. Open "dcli" in interactive mode.
   
   dcli +i
   
   
   
   
4. Execute below commands in dcli to list all the offline and online depots and confirm that the Depot ID noted in step (2) is present in either of the list.
   
   com vmware esx settings depots offline list

com vmware esx settings depots online list
   
   
5. Use dcli command to dump the contents of the depot using the Depot ID.
   
   
   1. If the missing Depot ID was found in the offline list, use the below command:
      
      com vmware esx settings depots offline content get --depot <depot ID>
      
      
   2. If the missing Depot ID was found in the online list, use the below command:
      
      com vmware esx settings depots online content get --depot <depot ID>
      
      
6. In the error case, the content will be different from what is being expected. For example shown in the issue/introduction section, the content of the depot does not match the WCP depot. The depot content resembles an ESXi depot than a WCP depot. If the depot content for a depot ID is not matching the expected depot then it can be concluded that the metadata has been overwritten and proceed with below steps to delete the depot.
   

   dcli> com vmware esx settings depots offline content get --depot B0B9326BD84BA41BCE40538B89669E243D6C707E
metadata_bundles:
vmw:
- hardware_support:
addons:
file_name: metadata-133.zip
solutions:
independent_components:
base_images:
- display_version: 9.0.1.0.########
display_name: ESXi
version: 9.0.1-0.########

   updates:
Broadcom-ELX-lpfc_900.14.4.390.20-36vmw.901.0.########:
summary: Broadcom Emulex Connectivity Division lpfc driver for FC adapters
Broadcom-bcm-mpi3_8.11.6.0.0.0-1vmw.900.0.########:
summary: Broadcom Native Data RAID
...
...

7. Delete the offending depot ID using dcli.
   
   com vmware esx settings depots offline delete --depot <depot ID>
   
   Note: If any error is seen during depot deletion, add "--force true" to the above command to force delete the depot.
   
   
8. Restart updatemgr service and wait for couple of minutes to stabilize.
   
   service-control --restart updatemgr
   
   
9. Retry the workflow that failed which may include reimporting the necessary depots into vLCM.