Broadcom Support routinely requests diagnostic information or a support bundle when a support request is handled. This diagnostic information contains logs and configuration files for your virtual machines.
This article provides the procedures for obtaining this diagnostic information.
The Endpoint Security solution consists of 3 primary components:
Thin Agent
Linux Thin Agent is an user space daemon which runs as a system service. t’s a multi-threaded application. Thin agent runs only in privileged mode. It leverages Linux kernel Fanotify calls and register itself to intercept file events and passes them to EPSecLib through the Mux. The Thin agent communicates with the Mux over VMCI (a proprietary host-only communication mechanism similar to IP).
Mux
The Mux is a ESXi User World component (analogous to a Unix process) which passes events from the Thin Agent to SVMs using EPSecLib. The Mux gets its configuration through REST API calls to the vShield Manager. Communication from the Mux to EPSecLib is done over TCP/IP. The Mux is essentially just another driver that is installed on the ESXi host.
EPSecLib
EPSeclib is a library used by partner solution SVMs to receive events from Thin Agents and perform operations (read/modify) on those files. EPSecLib can also block file operations. The EPSecLib exists as a virtual machine which is deployed by NSX and runs on on each ESXi host that is prepared for Endpoint. This virtual machine is called the SVA (Security Virtual Appliance).
General flow of a Guest introspection File scan:
Thin Agent
If a particular virtual machine is slow for file read and write operations, you find that its slow to unzip files, save files or perform backup jobs with a particular virtual machine, then you may be having issues specifically with the Thin agent.
When Troubleshooting Endpoint, the first thing you should check would be the compatibility of all the components involved. Compatibility is one of the main issues with Endpoint. You need the build numbers for ESXi, vCenter Server, NSX Manager, and which ever Security solution you have chosen (Trend Micro, McAfee, Kaspersky, Symantec etc). Once all of this data has been collected, you can compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrixes.
After ensuring that every VMware component works, check the Partner Compatibility Matrix to determine full compatibility.
Note: If you cannot find your Vendor compatibility information listed above, create a support ticket with your vendor.
Ensure that File Introspection is installed on the system.
Verify that thin agent is running by running service vsepd status. Once this command is executed you should see the vsep service in running state.
If you believe that the Thin Agent is causing a performance issue with the system, stop the service by running the command:service vsepd stop
service vsepd start
If you do verify that there is a performance problem with the Thin agent, please report to Broadcom with all the required logs collected.
You can enable Debug logging for the Linux Thin agent.
To turn on full logging:
/etc/vsep/vsep.conf
fileDEBUG_LEVEL=4
to DEBUG_LEVEL=7
for all logsDEBUG_LEVEL=6
for moderate logsDEBUG_DEST=2
) is vmware.log (on host) to change it to guest (i.e /var/log/message
or /var/log/syslog)
set DEBUG_DEST=1
The thin agent periodically checks for this level and logs accordingly. Hence there is no need to restart the thin agent after changing the log level.
Note: It is advised to not enable full logging unless absolutely necessary, as this can result in a heavy log activity flooding the vmware.log file. Also the size of vmware.log file can potentially grow to be very large. Hence please exercise caution while enabling full logging and disable it as soon as done.
Mux
If you see that all virtual machines on an ESXi host are not working with Endpoint, or you see alarms on a particular host regarding communication to the SVA, then it could be a problem with the MUX module on the ESXi host.
Check to see if the service is running on the ESXi host by running this command:# /etc/init.d/vShield-Endpoint-Mux status
For example:# /etc/init.d/vShield-Endpoint-Mux status
vShield-Endpoint-Mux is running
If you see that the service is not running, you can restart it or start it with this command:/etc/init.d/vShield-Endpoint-Mux start
or/etc/init.d/vShield-Endpoint-Mux restart
Note: It is safe to restart this service during production hours as it does not have any great impact, and restarts in a couple of seconds.
If you want to get a better idea of what the Mux module is doing or check the communication status, you can check the logs on the ESXi host. Mux logs are written to the host /var/log/syslog file. This is also included in the ESXi host support logs.
For more information, see Collecting diagnostic information for ESX/ESXi hosts and vCenter Server using the vSphere Web Client (2032892).
The default logging option for Mux is info and can be raised to debug to gather more information:
For more information, see Collecting diagnostic information for the NSX Guest Introspection MUX VIB (2094267).
Re-installing the Mux module can also fix many issues especially if the wrong version is installed, or the ESXi host was brought into the environment which previously had Endpoint installed on it. This needs to be removed and re-installed.
To remove the VIB, run this command: esxcli software vib remove -n epsec-mux
Note: You must reboot the ESXi host for this change to take effect. After the ESXi host has been rebooted, re-prepare the host again for Endpoint.
If you run into issues with the VIB installation, check the /var/log/esxupdate.log
file on the ESXi host. This log shows the most clear information as to why the driver did not successfully get installed. This is a common issue for Mux installation issues. For more information, see Installing NSX Guest Introspection services (MUX VIB) on the ESXi host fails in VMware NSX for vSphere 6.x (2135278).
Another common reason for an installation failure is a corrupt ESXi image. If this is the case:
Look for an error message similar to:esxupdate: esxupdate: ERROR: Installation Error: (None, 'No image profile is found on the host or image profile is empty. An image profile is required to install or remove VIBs. To install an image profile, use the esxcli image profile install command.')
You can verify if there is corruption:
cd /vmfs/volumes
on the ESXi host.find * | grep imgdb.tgz
Note: This command normally results in two matches.ls -l match_result
For example:> ls -l 0ca01e7f-########-####-##########/imgdb.tgz -rwx------ 1 root root 26393 Jul 20 19:28 0ca01e7f-########-####-##########/imgdb.tgz
> ls -l edbf587b-########-####-##########/imgdb.tgz -rwx------ 1 root root 93 Jul 19 17:32 edbf587b-########-####-##########/imgdb.tgz
imgdb.tgz
file is far greater than the other file or if one of the files is only a couple of bytes, it indicates that the file is corrupt. The only supported way to resolve this is to re-install ESXi for that particular ESXi host.EPSecLib
The NSX Manager handles the deployment of this virtual machine. In the past (with vShield), the third party SVA solution handles the deployment. That solution now connects to the NSX Manager. The NSX Manager handles the deployment of this SVA. If there are alarms on the SVA's in the environment, try and re-deploy them through the NSX Manager.
Notes:
SVA deployment and relationship between NSX and vCenter Server Process
eicar.test
.X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*.
/usr/sbin/vsep -v
will give the production version
Build number
------------------
Ubuntu
dpkg -l | grep vmware-nsx-gi-file
SLES12 and RHEL7
rpm -qa | grep vmware-nsx-gi-file
NSX for vSphere version.
esxcli software vib list | grep epsec-mux
.See these links for more information:
These sections describe how to isolate and troubleshoot specific issues.
In general, NSX GI partners provide the first level of technical support. VMware recommends contacting the partner where possible, particularly for performance and interoperability issues.
If thin agent crashes its core file will be generated at / directory. Collect the core dump file (core) from location / directory.
Note: Please use “file” command to check if core is generated by vsep.
For example:
# file core
core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/sbin/vsep'
Virtual machine crash
Enable the crashdump on linux system using the vendor specific method published by Ubuntu 14.04 TLS, Rhel 7 and Sles 12 respectively. After crashdump is enabled, system crash dump would be saved at the location specified by the vendor.
Collect the VMware vmss file of the virtual machine in a suspended state, see Suspending a virtual machine on ESX/ESXi to collect diagnostic information (2005831) or crash the virtual machine and collect the full memory dump file. VMware offers a utility to convert an ESXi vmss file to a core dump file.