Cisco Appliance VM Fails to Start Services After Migration ("systemkey" Error)

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• The Virtual Machine operating system (ADE-OS) boots successfully.
• Application services (e.g., ISE Application Server, Database) remain in a stopped or "Initializing" state.
• The VM console displays the following error during the boot sequence:
  Starting systemkey ... startup failed rc 1!
• The administrator may have selected "I Copied It" when powering on the VM in the new environment.

Cause

Cisco ADE-OS based appliances bind their internal database encryption and licensing to the SMBIOS UUID of the Virtual Machine.

When a VM is moved or cloned and the option "I Copied It" is selected in vCenter, VMware automatically generates a new UUID to avoid conflicts. The Cisco application detects this change, causing the decryption of the systemkey to fail because the hardware identity no longer matches the encrypted key stored on the disk.

Resolution

Option 1:

Revert via vCenter Advanced Settings

1. Power Off the Virtual Machine.
2. Right-click the VM and select Edit Settings.
3. Navigate to VM Options > Advanced > Configuration Parameters > Edit Configuration.
4. Locate the parameter uuid.bios.
5. Change the value to match the original UUID (from the source VM or backup .vmx file).
   • Note: If the original UUID is lost, you may need to redeploy the appliance and restore from a backup.
6. Click OK to save.
7. Power On the VM.

Option 2: Edit the .vmx File Directly

1. Power Off the VM.
2. SSH to the ESXi host the VM is registered to via root
3. Obtain the Inventory ID (Vmid) for the virtual machine
   vim-cmd vmsvc/getallvms | grep -i <vm-name>
4. Change your working directory to VM folder
   cd /vmfs/volumes/<datastore_name>/vmfolder/
5. Open the file in a text editor.
   vi vm-name.vmx
6. Press i (for insert option)
7. Find the line starting with uuid.bios = "...".
8. Replace the UUID string with the original value.
9. Save the file and exit (press ESC, type :wq!, press Enter).
10. Reload the VM configuration Reloading a vmx file without removing the virtual machine from inventory
11. Power on the virtual Machine.

Additional Information

Changing or keeping a UUID for a moved virtual machine

Reloading a vmx file without removing the virtual machine from inventory