VMware NSX Service nsx-context-mux is not running on ESXi 8.0.3 impacting services such as IDFW, IDS/IPS, EPP, MPS.

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention
• VMware NSX Firewall

Issue/Introduction

• VMware NSX 4.1.2.1 or earlier is in use.
• ESXi hosts were upgraded to version 8.0.3.
• NSX features such as IDFW, IDS/IPS, EPP, MPS are impacted.
  • IDFW rules stop working if the Guest Introspection (GI) is configured as the only method of logon detection.

    • ESXi host STOPS detecting logons (Below is the example of a detection, a working scenario for GI based detection)

       [root@ESXi:~] egrep "SESSION_TYPE_CONNECT|SESSION_TYPE_LOGON|SESSION_TYPE_DISCONNECT" /var/run/log/nsx-syslog.log

       2024-07-09T14:06:05.547Z In(182) nsx-opsagent[526613]: NSX 526613 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="ctxteng" tid="526932" level="INFO"] Context: Session dump - vcUuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, dfwKey: , sid: , uid: -1, type: SESSION_TYPE_CONNECT, user name: , domain name: , session id: 4, client ip: , ip version: 65535, timestamp: 1720533xxxxxx, group count: 0, group hash: 0

       2024-07-09T14:06:44.473Z In(182) nsx-opsagent[526613]: NSX 526613 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="ctxteng" tid="526928" level="INFO"] Context: Session dump - vcUuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, dfwKey: S-1-5-21-xxxxxxxxx-xxxxxxxxx-xxxxxxxx-xxx, sid: S-1-5-xx-xxxxxxxxxx-xxxxxxxxx-xxxxxxxx-xxx, uid: -1, type: SESSION_TYPE_CONNECT, user name: Administrator, domain name: MYIDFWDOMAINXX, session id: 2, client ip: , ip version: 65535, timestamp: 172053400xxxx, group count: 15, group hash: 0

  • IDS/IPS and/or MPS illustrates below error in UI when attempted to enable rules.
    • [Error Code = '9001', Error Message = 'Distributed MPS config connection failure.', Affected Entities = '[]'.]

• The nsx-context-mux service is not running on the ESXi:

  • [root@ESXi:~] /etc/init.d/nsx-context-mux status

    nsx-context-mux is not running

• Attempted to start the service results it in going back to a not running state:

  • [root@ESXi:~] /etc/init.d/nsx-context-mux start

    mux_user already exists. Not recreating

    userName = mux_user for namespace access

    Unable to add rp for Mux

    nsx-context-mux started                             <<<<<<<<<< Service starts

    [root@ESXi:~] /etc/init.d/nsx-context-mux status

    nsx-context-mux is not running                  <<<<<<<<<< Goes back to stopped state.

 

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX

vDefend Firewall

vDefend Firewall with Advance Threat Prevention

Cause

The issue is due to the mismatch in the Python library installed on the ESXi host.

The nsx-context-mux requires Python 3.8 for NSX version 4.1.2.1 and lower. But from ESX 8.0.3, python 3.11 is in use. 

You can confirm the version of python installed on the ESXi :

[root@ESX:~] cd /lib64/
[root@ESX:/lib64] ls -l | grep libpython
lrwxrwxrwx    1 root     root            20 Jun 11  2024 libpython3.11.so -> libpython3.11.so.1.0               <<<<<<<<<<<<<<< we see that python 3.11 is being used here
-r-xr-xr-x    1 root     root       4824600 Jun 11  2024 libpython3.11.so.1.0
-r-xr-xr-x    1 root     root         15536 Jun 11  2024 libpython3.so

Resolution

This issue is resolved in NSX 4.1.2.3 and above. Broadcom recommends upgrading NSX to a recent release.

Additional Information

Workaround: For IDFW use Event Log Scraping (ELS) for login/logout event instead of GI.