Unexpected error while upgrading upgrade unit: Upgrade not attempted because dry-run of upgrade on host [UUID]failed with message: [LockingError] Another process is updating the ESX image

Products

• VMware NSX
• VMware Live Recovery

Issue/Introduction

• NSX upgrade of Transport node fails with error:
  
  Unexpected error while upgrading upgrade unit: Upgrade not attempted because dry-run of upgrade on host [UUID] failed with message: [LockingError] Another process is updating the ESX image. Please try again later. Please refer to the log file for more details..
  
  
• vSphere Replication is installed in the environment.
• During NSX Transport Node upgrade the ESX logs show vSphere Replication agent (HBR) is also attempting an upgrade.
  
  /var/log/esxupdate.log
[Timestamp] In(14)[+] esxupdate[12345678]: Command: update
[Timestamp] In(14)[+] esxupdate[12345678]: Args: ['update']
[Timestamp] In(14)[+] esxupdate[12345678]: Options:
[Timestamp] In(14) esxupdate[12345678]: viburls = ['https://[FQDN]:443/vib/vmware-hbr-agent.vib']
  
  
• ESX shows the NSX upgrade process attempting to access the ESX image.
  
  /var/log/esxupdate.log
[Timestamp] In(14) esxupdate[123456789]: Installers initiated are {'quickpatch': <vmware.esximage.Installer.QuickPatchInstaller.QuickPatchInstaller object at [HEX]>, 'live': <vmware.esximage.Installer.LiveImageInstaller.LiveImageInstaller object at [HEX]>, 'boot': <vmware.esximage.Installer.BootBankInstaller.BootBankInstaller object at [HEX]>, 'locker': <vmware.esximage.Installer.LockerInstaller.LockerInstaller object at [HEX]>}
[Timestamp] Er(11) esxupdate[123456789]: Error locking file /var/run/esximg.pid: [Errno 11] Resource temporarily unavailable, the file is currently locked by process with PID 12345678
[Timestamp] Wa(12) esxupdate[12345678]: Failed to acquire lock: ('/var/run/esximg.pid', 'Error locking file /var/run/esximg.pid: [Errno 11] Resource temporarily unavailable, the file is currently locked by process with PID 12345678')
  
  
• Without making any changes, retrying the host upgrade may succeed without issue. Subsequent hosts may fail with the same error.

Environment

VMware NSX
VMware Replication

Resolution

This is a known issue impacting VMware NSX and VMware Replication.

Workaround:

1. SSH to vSphere Replication Appliance.
2. Run the following command to disable the HBR VIB upgrade: /opt/vmware/hms/bin/hms-configtool -cmd reconfig -property hms-auto-install-hbragent-vib=false   
3. On the replication appliance, restart the HMS service: systemctl restart hms
4. Complete the Transport Node upgrade.
5. Re-enable HBR VIB upgrade: /opt/vmware/hms/bin/hms-configtool -cmd reconfig -property hms-auto-install-hbragent-vib=true
6. Restart HMS service as per Step 3.