MUX vib removal fails with 'Error in running rm /tardisks/epsec_mu.v00'

Products

• VMware vDefend Firewall

Issue/Introduction

Symptoms:
Removing Guest Introspection deployment fails. In the eam.log you see the following error:

2018-03-20T07:37:42.797Z | INFO | vim-async-0 | VimLRO.java | 62 | 'Task:task-5049:b73e2361-ccfd-40ee-b2d2-643e721c9589' completed with result vim.host.PatchManager.Result {
   xmlResult = '<esxupdate-response>
<version>1.50</version>
<error errorClass="LiveInstallationError">
  <errorCode>29</errorCode>
  <errorDesc>Live vib installation failed. An immediate reboot is required to clear live vib installation failure.</errorDesc>
  <vibs>[]</vibs>
  <msg>([], "Error in running rm /tardisks/epsec_mu.v00:\nReturn code: 1\nOutput: rm: can't remove '/tardisks/epsec_mu.v00': Device or resource busy\n\nIt is not safe to continue. Please reboot the host immediately to discard the unfinished update.")</msg>
</error>
</esxupdate-response>
',
   version = '1.50',
   status = <unset>,
}
2018-03-20T07:37:42.797Z | INFO | host-24-4 | VcPatchManager.java | 202 | Uninstall result on VcHostSystem(ID: host-24) for [epsec-mux:6.5.0esx65-7899169]:
<esxupdate-response>
<version>1.50</version>
<error errorClass="LiveInstallationError">
  <errorCode>29</errorCode>
  <errorDesc>Live vib installation failed. An immediate reboot is required to clear live vib installation failure.</errorDesc>
  <vibs>[]</vibs>
  <msg>([], "Error in running rm /tardisks/epsec_mu.v00:\nReturn code: 1\nOutput: rm: can't remove '/tardisks/epsec_mu.v00': Device or resource busy\n\nIt is not safe to continue. Please reboot the host immediately to discard the unfinished update.")</msg>
</error>
</esxupdate-response>

Environment

VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.3.x

Cause

As the above error indicates, there is a problem with the MUX deamon, that cannot be stopped with the vShield-Endpoint-Mux script.

Resolution

Edit the vShield-Endpoint-Mux script on all the hosts and add a force flag to the pkill command:
 

1. Copy the file to an editable location

cp /etc/init.d/vShield-Endpoint-Mux /vShield-Endpoint-Mux-edit

 

2. Use VI to edit the file 

vi vShield-Endpoint-Mux-edit

change:
pkill -KILL "${VSEPMUX}"

to:
pkill -KILL -f "${VSEPMUX}"

 

3. Verify the change took place (look for the -f)

less vShield-Endpoint-Mux-edit

 

4. Move the edited file to the correct location

mv vShield-Endpoint-Mux-edit /etc/init.d/vShield-Endpoint-Mux

 

5. Verify the file was moved (look for the -f)

less /etc/init.d/vShield-Endpoint-Mux

Then try again.