HCX Migrations fail with error "The session is not authenticated"

Products

• VMware HCX

Issue/Introduction

• HCX migrations are randomly failing with the error "The session is not authenticated". 
  
  
  
• The migration had already completed "initial sync" and was in "waiting for maintenance window" status.
  
  
• Reviewing HCX-MGR app.log found in /common/logs/admin shows:
  <timestamps> UTC [ReplicationTransferService_SvcThread-41527, Ent: HybridityAdmin, , TxId: ########-####-####-####-145f5b0d717f] ERROR c.v.v.h.m.common.MigrationUtil- [migId=null] Job (########-####-####-####-8f07d9b4665e) failed with exception The session is not authenticated.com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.
<timestamps> UTC [ReplicationTransferService_SvcThread-41527, Ent: HybridityAdmin, , TxId: ########-####-####-####-145f5b0d717f] ERROR c.v.h.s.r.ReplicationTransferService- Error logging into vC due to invalid credentials, retrying the ReplicationTransferSourceCleanupJob with Id ########-####-####-####-8f07d9b4665e.
  
  
• Tracking the TxId from above log messages in vCenter /var/log/vmware/vpxd/vpxd.log - shows "missed heartbeats" for multiple hosts:
  <timestamps> info vpxd[10990] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-ac06256] VPXA heartbeat build initialized; [vim.HostSystem:host-####,############.###.####.#############.###], msg: {srv: #######, gen: 778759, ct: 803153, bld: 24585383, cnx: ########-####-####-####-36258496f0cc, ip: ##.###.##.##}
<timestamps> info vpxd[10990] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-ac06256] Missed heartbeats for host; [vim.HostSystem:host-####,############.###.####.#############.###], missed: 803152, msg: {srv: #######, gen: 778759, ct: 803153, bld: 24585383, cnx: ########-####-####-####-36258496f0cc, ip: ##.###.##.##}

Environment

VMware HCX

Cause

• HCX connectivity to vCenter was impacted due to an external networking event.
• HCX must have a reliable consistent communication to vCenter while migrations are ongoing. Any connectivity loss to vCenter will cause the HCX_Migration_Tracker workflow and subsequently the migration to fail.

Resolution

• Review and resolve the underlying infrastructure issue affecting HCX -> VC connectivity..
• HCX 4.11.1 provides additional resiliency to such disconnect events between HCX and VC.
  • This enhancements are time-bound. If the vCenter endpoint remains unreachable for more than 2 to 2.5 hours, ongoing migrations will fail.
  • Please read: vCenter Connection Enhancements
• Once the connectivity between vCenter and  HCX manager is stable, reinitiate the failed migrations.

Additional Information

HCX 4.11.1 Release Notes