NSX Endpoint Security Multiplexer (EPSEC-MUX) Interfering with ESXi Host Upgrade Process
search cancel

NSX Endpoint Security Multiplexer (EPSEC-MUX) Interfering with ESXi Host Upgrade Process

book

Article ID: 381495

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

During the installation or upgrade of ESXi hosts to version 8.0, the following error message may be encountered:

These VIB(s) on the host do not have the required sha-256 checksum for their payloads: VMware_bootbank_epsec-mux_6.5.0esx60-7357581 This will prevent VIB security verification and secure boot from functioning properly. Please remove these VIBs and check with your vendor for a replacement of these VIBs.

Environment

VMware ESXi 

VMware NSX 

Cause

The EPSEC-MUX (Endpoint Security Multiplexer) module is not in use, as the NSX environment has not been set up. However, the outdated EPSEC-MUX VIB (version 6.5.0esx60-7357581) still resides on the ESXi host, causing conflicts during the upgrade process.

Resolution

1.Verify the presence of the outdated EPSEC-MUX VIB on this ESXi host:

  • Log in to the ESXi host as root.Check if the EPSEC-MUX VIB is installed using the following command:

          esxcli software vib list | grep epsec-mux

  • Check the status of the vShield-Endpoint and NSX-Context-Engine services:
  • Use the following command to check the service status:

    /etc/init.d/vShield-Endpoint-Mux status
  • Example output:

[root@esxihost:/etc/init.d] ./vShield-Endpoint-Mux status
vShield-Endpoint-Mux is not running
NSX-Context-Engine is running

2. Remove outdate VIB using below command 

     esxcli software vib remove -n epsec-mux

If removing VIB failed and throws error as below:

3. The stop the vShield-Endpoint-Mux and NSX-Context-Engine services if they are running:

  • To stop the vShield-Endpoint-Mux service, run:

          /etc/init.d/vShield-Endpoint-Mux stop

  • To stop the NSX-Context-Engine service, use:

    /etc/init.d/vShield-Endpoint-Mux stop-nsx-ctxteng

4 .If the services cannot be stopped, check if the following daemons are enabled at the runlevel:

  • Run the following command to check:

        chkconfig --list

  •  If watchdog-vShield-Endpoint-Mux and watchdog-NSX-Context-Engine daemons are enabled (or if only the vShield-Endpoint-Mux daemon is enabled), turn them off using the following   command:
    chkconfig vShield-Endpoint-Mux off

5. And then Stop the vShield-Endpoint-Mux and NSX-Context-Engine services again:

  • After disabling the daemons, this will allow to stop these services now:

        /etc/init.d/vShield-Endpoint-Mux stop
       /etc/init.d/vShield-Endpoint-Mux stop-nsx-ctxteng

6. Additionally, check if any Mux processes are currently running:
 

  • Run the following command to verify if the vShield-Endpoint-Mux processes are active on the ESXi host:

         # ps -c | grep Mux
         192223 192223 sh /bin/sh /sbin/watchdog.sh -s vShield-Endpoint-Mux -q 100 -t 1000000 /usr/lib/vmware/vShield-Endpoint-Mux 900 -c 910
         192233 192233 vShield-Endpoint-Mux /usr/lib/vmware/vShield-Endpoint-Mux 900 -c 910
         192236 192233 vShield-Endpoint-Mux /usr/lib/vmware/vShield-Endpoint-Mux 900 -c 910

       If any Mux processes are running, use the following command to kill the processes (including the watchdog.sh process):

        # kill -9 192223 192233 192236

7. Once the services are stopped, remove the outdated EPSEC-MUX VIB using the following command:

    esxcli software vib remove -f -n epsec-mux --no-live-install


8. After the VIB is removed, confirm that it's no longer present on the ESXi host:
    esxcli software vib list | grep epsec-mux

 

[Note: All these steps on the ESXi host command shell with root login.]

 

 

Additional Information

In this case, since NSX is not being used, stopping the vShield-Endpoint-Mux service and removing the EPSEC-MUX VIB is not an issue. However, for customers who are actively using NSX, may require further investigation on obtaining a replacement EPSEC-MUX VIB with the correct checksum .

Kindly Ref below docs for more details:

ESX EPP Module (MUX) Logs

Troubleshooting ESX GI Module (MUX)

 KB318056

Collecting diagnostic information for the NSX Guest Introspection MUX VIB

https://github.com/lamw/ghettoVCB/issues/275