Recovered Virtual Machines Automatically Unregister When Using Pure Storage vSphere Plugin

Products

• VMware vCenter Server

Issue/Introduction

• During the recovery of VMs exceeding 1TB, leaving the CBT prompt open during the storage migration resulting in the VM being unregistered upon completion of the migration.
  
  
  
  
• On smaller VMs the migration finishes first, so the cleanup has nothing to collide with and everything works normally
  
  
• An entry in /var/log/vmware/vpxd/vpxd.log shows,

     YYYY-MM-DDTHH:MM:SS INFO vpxd 64153 [vc@4413 sub="vpxLro" opID="5#####b"] [VpxLRO] – BEGIN task-#### – vm-#### – vim.VirtualMachine.relocate – 52###5-d##d-a##7-8##1-bc3#####5fcbb(52####70-5##7-9##5-c##c-8d####0598)
     YYYY-MM-DDTHH:MM:SS INFO vpxd 64129 [vc@4413 sub="vpxLro" opID="17cef340"] [VpxLRO] – BEGIN lro-#### – SessionManager – vim.SessionManager.cloneSession – 52####d8-0##8-a##6-e##5-ea###090ce
     YYYY-MM-DDTHH:MM:SS INFO vpxd 64117 [vc@4413 sub="vpxLro" opID="4503841"] [VpxLRO] – BEGIN task-### – vm-#### – vim.VirtualMachine.unregister – 52###5-d##d-a##7-8##1-bc3#####5fcbb(52####70-5##7-9##5-c##c-8d####0598)
     YYYY-MM-DDTHH:MM:SS INFO vpxd 64120 [vc@4413 sub="vpxLro" opID="6589ee21"] [VpxLRO] – BEGIN task-### – vm-#### – vim.VirtualMachine.unregister – 52###5-d##d-a##7-8##1-bc3#####5fcbb(52####70-5##7-9##5-c##c-8d####0598)

• An entry in /var/run/log/hostd.log shows,

        YYYY-MM-DDTHH:MM:SS -DEBUG Hostd 2099759 [esx@4413 sub="Vmsvc.vm:/vmfs/volumes/6####4b-4###99-d##c-002###012/<NAME_OF_VM>/NAME_OF_VM.vmx" opID="5#####b-01-##-d909" sid="52###79b" user="vpxuser:####"] Unregister called on virtual machine
    YYYY-MM-DDTHH:MM:SS -INFO Hostd 2099767 [esx@4413 sub="Vimsvc.TaskManager" opID="5#####b-01-1-d91d" sid="5267979b" user="vpxuser:####"] Task Created : haTask-ha-folder-vm-vim.Folder.createVm-17###13
    YYYY-MM-DDTHH:MM:SS -INFO Hostd 2099757 [esx@4413 sub="Vimsvc.TaskManager" opID="5#####b-01-1-d91d" sid="5267979b" user="vpxuser:#####"] Task Completed : haTask-ha-folder-vm-vim.Folder.createVm-1###13 Status success
    YYYY-MM-DDTHH:MM:SS -DEBUG Hostd 2099763 [esx@4413 sub="Vmsvc.vm:/vmfs/volumes/6####4b-4###99-d##c-002###012/<NAME_OF_VM>/NAME_OF_VM.vmx" opID="5#####b-01-1-d994" sid="5267979b" user="vpxuser:#####"] Destroy VM complete

• An entry in/var/log/vmware/envoy/envoy-access.log shows,

          YYYY-MM-DDTHH:MM:SS info envoy[61262] POST /sdk 200 via_upstream - 2ms 1ms <IP_ADDRESS_PURE_STORAFE_PLUGIN>:48212 721b 0ms TLSv1.3 HTTP/1.1 <IP_ADDRESS_VCENTER>:443 0ms 1 127.0.0.1:59218 582b  0ms -HTTP/2 93+471b 82+819+0b 910b 0ms 127.0.0.1:8085 "CloneSession"
     YYYY-MM-DDTHH:MM:SS info envoy[61262] POST /sdk 500 via_upstream -<IP_ADDRESS_PURE_STORAFE_PLUGIN>:48230 728b 0ms  <IP_ADDRESS_VCENTER>:443 0ms 1 127.0.0.1:59218  33b  0ms - HTTP/2 109+406b 127.0.0.1:8085 - - "UnregisterVM"
     YYYY-MM-DDTHH:MM:SS info envoy[61262] POST 200 via_upstream <IP_ADDRESS_PURE_STORAFE_PLUGIN>:35614 728b 0ms TLSv1.3 HTTP/1.1 496+406b 118+390+0b<IP_ADDRESS_VCENTER>:443 0ms 1 127.0.0.1:59204 5 4b  0ms - HTTP/2 90+406b 13+390+0b 412b 5ms 127.0.0.1:8085 "UnregisterVM"
     YYYY-MM-DDTHH:MM:SS info envoy[61262] POST /sdk 200 via_upstream <IP_ADDRESS_PURE_STORAFE_PLUGIN>:40846 551b 0ms TLSv1.3 HTTP/1.1  <IP_ADDRESS_VCENTER>:443 0ms 1 127.0.0.1 337b 0ms -HTTP/2 88+231b 13+443+0b 465b 0ms 127.0.0.1:8085 - - "PowerOffVM_Task"
     YYYY-MM-DDTHH:MM:SS info envoy[61262] POST /sdk 200 via_upstream - 3ms 3ms <IP_ADDRESS_PURE_STORAFE_PLUGIN>:40846 620b 0ms TLSv1.3 HTTP/1.1 494+300b  <IP_ADDRESS_VCENTER>:443 0ms 1 127.0.0.1:59218 406b  0ms - HTTP/2 88+300b  127.0.0.1:8085 - - "RelocateVM_Task

Environment

vCenter server 9.1.1

Pure storage plugin 5.5.1

Cause

The issue is caused by the Pure Storage vSphere plugin server initiating redundant UnregisterVM API calls during the VM recovery workflow. Because vCenter Server serializes API operations on a per-VM basis, these requests are queued by the vCenter task manager while the legitimate VM relocation (Storage vMotion) is still in progress. Once the relocation task finishes, vCenter immediately executes the queued unregister command, removing the newly restored virtual machine from the vCenter inventory.

Resolution

Engage Pure Storage Support for assistance