ATP VE fails to serve UI after ESXi host loses connection to its remote datastore

book

Article ID: 164784

calendar_today

Updated On:

Products

Endpoint Detection and Response Advanced Threat Protection Platform

Issue/Introduction

The management server of Advanced Threat Protection (ATP) Virtual Edition (VE) fails to serve a user interface (UI) after the ESXi host where ATP VE is deployed loses connection to its remote datastore.

Instead of reaching the user interface, the client browser will search the internet or display "Unable to connect".

If support is contacted while ATP has its /tmp partition in read-only mode, and before the machine is restarted, support can find messages similar to the following in /var/log/dmesg:

[36764.170719] sd 0:0:0:0: [sda] SCSI device reset on scsi0:0
[36785.900867] sd 0:0:0:0: timing out command, waited 180s
...
[36785.903962] EXT4-fs warning (device sda8): ext4_end_bio:332: I/O error -5 wri               ting to inode 1705439 (offset 0 size 0 starting block 115391320)
...
[37150.529804] Buffer I/O error on device sda8, logical block 0
[37150.530810] lost page write due to I/O error on sda8
...
[37150.531832] EXT4-fs (sda8): Remounting filesystem read-only
[37150.531835] EXT4-fs (sda8): previous I/O error to superblock detected
[37150.535925] Detected aborted journal

 

 

Cause

  1. The ESXi host lost connection to the NAS device functioning as its datastore.
  2. After repeated attempts to write to multiple partitions that are usually mounted as read-write partitions, ATP marked those partitions as read-only.
  3. During the time the /tmp partition was marked as read-only, multiple attempts to write updates for events being processed and configuration data failed, resulting in corrupted state and configuration data for ATP VE.

Environment

  • ATP Virtual Edition is deployed from ova to an ESXi host.
  • The datastore for ESXi host is a Network Attached Storage (NAS) device rather than a physical drive local to the ESXi host.

Resolution

To repair the corruption caused by the ESXi Host losing contact with its datastore, please re-install ATP from OVA.