vSAN 6.7 P03-P05 or vSAN 7.0 U1/U2 cluster experiences inaccessible objects or unexpected object deletion

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Inaccessible objects
• Unexpected object deletion

Environment

VMware vSAN 6.7.x
VMware vSAN 7.0.x

Cause

The symptoms may occur in the following vSAN releases due to inconsistency of the EPD database.

Releases containing the defect:

• vSAN 6.7P03-P05
• vSAN 7.0 U1/U2

EPD (Entry Persistence Daemon) is a user space daemon that runs on every host that is part of the vSAN cluster.  The main job of EPD is to make sure there is no component leakage when objects are deleted.

The defect is due to the SQLite database used by EPD being damaged and results in objects becoming inaccessible or deleted unexpectedly.

Resolution

Patch up to fixed release as soon as possible.

Fixed releases:

• vSAN 6.7 P06+
• vSAN 7.0 U3c+

Workaround:
Run the following command on each host in the cluster:
/etc/init.d/epd stop; rm -rf /scratch/epd-store*.lock; python -c "import sqlite3; conn = sqlite3.connect('/scratch/epd-storeV2.db'); conn.execute('DELETE FROM cmmds_entries;'); conn.commit()"
This does the following:

• Stops EPD service on all hosts
• Delete epd-storeV2.db on all hosts

Note: DO NOT START EPD until host is patched up to a fixed build

Workaround will prevent further harm while hosts are patched up, but cannot be used for more than 24hrs
Once above command has been run, immediately patch hosts

Additional Information

Impact/Risks:

• Guests may become inaccessible
• Some objects may be deleted in error