Cannot Force Delete a Service Mesh

Products

• VMware HCX

Issue/Introduction

• The mesh has only the IX appliance which was removed abnormally.
• Mesh deletion is attempted and it fails due to migration that are not in a Draft or Archived state.
• A migration fails to archive with the error java.lan.NullPointerException.

• Login to the HCX manager as user admin and verify the log file /common/logs/admin/web.log, you may see the following.
  [https-jsse-nio-127.xxx.0.1-8443-exec-3, Ent: HybridityAdmin, , TxId: TxId: ########-####-####-#####-############] ERROR c.v.v.h.m.a.MigrationArchiveUtil- Encountered error archiving migration: ########-####-####-#####-############, migrationGroupId: ########-####-####-#####-############ java.lang.NullPointerException: null
  
  
• Force delete fails to remove the mesh.
  
  

Environment

VMware HCX 

Cause

• java.lang.NullPointerException occurs when a function calls for a variable who's value is null.
• Having a null value in this case means that the object does not exist.
• In this example the IX appliance was removed outside of HCX.
• When HCX is attempting to remove the SM it checks for the existence of all appliances in VC inventory.
  • If the appliance does not exists, the force cleanup workflow fails. 

Resolution

• Please open a service request with Broadcom support to document this issue. Please provide the below information:
  • Name of Service-Mesh affected.
  • Error seen in UI and actions taken to remove/cleanup the object.
  • Source/Target HCX Manager log bundle WITH DB dump option selected. 
• It might be possible to modify the database to forcefully remove the SM bits holding up the "force cleanup" workflow.
• NOTE: This process if very time intensive and may take several days or longer
• The quickest way to recover from this event is to redeploy both source/target HCX Manager instances.
  • If proceeding to remove/redeploy HCX Manager instances, be sure to remove the appliances from VC inventory.
    • This is the only time it is recommended to delete HCX appliances outside of HCX GUI.