Collecting diagnostic information for the NSX Guest Introspection MUX VIB

Products

• VMware vSphere ESXi
• VMware vDefend Firewall

Issue/Introduction

VMware Technical Support routinely requests diagnostic information or a support bundle when a support request is handled. This article provides the procedures for obtaining this diagnostic information for the ESXi host component (MUX) for NSX Guest Introspection.

 

The diagnostic information collected can then be uploaded to Broadcom Support.

Environment

VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.3.x
VMware vSphere ESXi 5.1
VMware NSX for vSphere 6.4.x
VMware vSphere ESXi 5.0
VMware vCloud Networking and Security 5.5.x
VMware vSphere ESXi 5.5
VMware NSX for vSphere 6.0.x
VMware NSX for vSphere 6.1.x

Resolution

Enabling Mux Logging

To view the version of the Mux VIB installed on the host, run this command:

#esxcli software vib list | grep epsec-mux.

 

To turn full log on:

Note: Peform these steps on the ESXi host command shell.

1. Run this command to find the Mux processes that are currently running:
   
   ~ # ps -c | grep Mux
   192223 192223 sh /bin/sh /sbin/watchdog.sh -s vShield-Endpoint-Mux -q 100 -t 1000000 /usr/lib/vmware/vShield-Endpoint-Mux 900 -c 910
   192233 192233 vShield-Endpoint-Mux /usr/lib/vmware/vShield-Endpoint-Mux 900 -c 910
   192236 192233 vShield-Endpoint-Mux /usr/lib/vmware/vShield-Endpoint-Mux 900 -c 910
    
2. Run this command to kill the running Mux processes, including the watchdog.sh process:
   
   ~ # kill -9 192223 192233 192236
   
   Note: There are two Mux processes that get spawned.
    
3. Start Mux with a new -d option.
   
   ~ # /usr/lib/vmware/vShield-Endpoint-Mux –d 900 –c 910
   
   Note: Option -d does not exist for epsec-mux builds 5.1.0-01255202 and 5.1.0-01814505

You can view the Mux log messages in the /var/log/syslog.log file on the ESXi host.

Enabling Debug logging on the Mux

1. To enable Debug for Mux, run this command:
   
   # echo ENABLE_SUPPORT=TRUE > /etc/muxconfig.conf; echo ENABLE_DEBUG=TRUE >> /etc/muxconfig.conf
    
2. Once complete, it should appear like this:
   
   # cat /etc/muxconfig.conf
   
   ENABLE_SUPPORT=TRUE
   ENABLE_DEBUG=TRUE

Additional Information

MuxConfig.xml File

This section details the location of key entries in the muxconfig.xml file and provides a sample muxconfig.xml file for reference.

XML file Location

The file is located in the /var/run/muxconfig.xml directory on the host.

Important entries in the XML file

Ensure that the entries corresponding to the global solutions, solution ID, and port number are specified correctly. See the sample muxconfig.xml file:

 

<?xml version="1.0" encoding="UTF-8"?>
 <EndpointConfig>
   <InstalledSolutions>
     <Solution>
       <id>100</id>
       <ipAddress>xxx.xxx.xxx.xxx</ipAddress>
       <listenOn>ip</listenOn>
       <port>48655</port>
       <uuid>42383371-xxxx-xxxx-xxxx-f1d9c52ab1d0</uuid>
       <vmxPath>/vmfs/volumes/7adf9e00-xxxxxxxx/EndpointService (216)/EndpointService (216).vmx</vmxPath>
     </Solution>
     <Solution>
       <id>102</id>
       <ipAddress>xxx.xxx.xxx.xxx</ipAddress>
       <listenOn>ip</listenOn>
       <port>48651</port>
       <uuid>423839c4-xxxx-xxxx-xxxx-79870da05291</uuid>
       <vmxPath>/vmfs/volumes/7adf9e00-xxxxxxxx/apoon/EndpointSVM-alpha-01/EndpointSVM-alpha-01.vmx</vmxPath>
     </Solution>
     <Solution>
       <id>6341068275337723904</id>
       <ipAddress>xxx.xxx.xxx.xxx</ipAddress>
       <listenOn>ip</listenOn>
       <port>48655</port>
       <uuid>42388025-xxxx-xxxx-xxxx-a143b9cbd1ee</uuid>
       <vmxPath>/vmfs/volumes/7adf9e00-xxxxxxxx/DlpService (1)/DlpService (1).vmx</vmxPath>
     </Solution>
   </InstalledSolutions>
   <DefaultSolutions/>
   <GlobalSolutions>   
     <solution>
       <id>100</id>
       <tag></tag>
       <order>0</order>
     </solution>
     <solution>
       <id>102</id>
       <tag></tag>
       <order>10000</order>
     </solution>
     <solution>
       <id>6341068275337723904</id>
       <tag></tag>
       <order>10001</order>
     </solution>
   </GlobalSolutions>
 </EndpointConfig>