Entropy VOBs introduced in vSphere 8.0

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

Following vSphere warnings or errors are observed on an ESXi host when a potential problems are detected in Entropy Daemon:

1. VOB ID: esx.audit.entropy.available.low

2. VOB ID: esx.problem.entropy.inmemory.empty

3. VOB ID: esx.problem.entropy.empty

4. VOB ID: esx.audit.entropy.external.source.disconnected

5. VOB ID: esx.problem.entropy.config.error

Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 8.0.1

Cause

1. Entropy available in memory cache is below low watermark.

2. Entropy available in memory cache is exhausted.

3. Entropy available in memory cache and storage cache is exhausted.

4. External entropy source disconnected.

5. Entropy config error.

Resolution

To resolve the issue, please follow the steps mentioned for each of the errors:

1. VOB ID: esx.audit.entropy.available.low:

Host is configured with external entropy source.  Host caches entropy bits in memory to serve crypto applications.  The host cache for external entropy is running low.
Please ensure the configured entropy source is pushing sufficient bits of entropy.
Possible reasons and resolution:

1. Not enough entropy bits  being sent by the client – increase the amount or frequency of the ADD api calls.  Refer Configure ESXi Entropy for api details.

2. There is disruption in network/infrastructure components pushing external entropy – Please ensure infrastructure is sized to support the entropy consumption appropriately.

3. In memory cache insufficient  – Consider increasing the memory cache size if you see these warnings repeatedly.
    

2. VOB ID: esx.problem.entropy.inmemory.empty

Host is configured with external entropy source. Host caches entropy bits in memory to serve crypto applications. The entropy available in the host’s memory cache is exhausted.
Please ensure the configured entropy source is pushing sufficient bits of entropy.
Possible reasons and resolution:

1. Not enough entropy bits  being sent by the client – increase the amount or frequency of the ADD api calls.  Refer Configure ESXi Entropy for api details.

2. In memory cache insufficient  – Consider increasing the memory cache size if you see these warnings repeatedly.

If entropy API is not reachable, Please reboot the host.
 

3. VOB ID: esx.problem.entropy.empty

Host is configured with external entropy source. Host caches entropy bits in memory as well as in storage to serve crypto applications. The entropy available in host’s  memory cache and storage cache is exhausted.
Please make sure the storage cache file with sufficient amount of entropy is provided when host boots up.
Possible reasons and resolution:

1. In memory cache or storage cache insufficient  – Consider increasing the storage cache or memory cache.

2. Interruption to external entropy source or external entropy source not available at early boot – Make sure external entropy source starts sending entropy as soon as the host boots up.

If entropy API is not reachable, there are two ways to recover:

A. If user is ok to tolerate other entropy sources for remediation, follow these steps to recover the system:

1. Perform hard reset on the host (Shutdown + Power on)

2. Upon power on, pause the boot process  by pressing "Shift+O" on the console.

3. Provide boot option "entropySources=0xF" and press enter. This will boot the host with multiple entropy sources. (Note that your box might be out of compliance during remediation as multiple entropy sources such are rdseed/interrupts are being used to provide random numbers during remediation.  If that is not acceptable, host should be re installed with external entropy)

4. Once host is booted up successfully, start supplying entropy to host by the ADD API to replenish the storage cache.

5. Ensure storage cache is populated. There are two ways to check if entropy in the storage cache is replenished:

6. Login to ESXi shell and check for "Entropy in storage cache is replenished completely" log in "/var/run/log/entropyd.log" file. Once this log comes in entropyd log file that means storage cache is replenished.

7. Call entropy GET API and check entropy available in host. If entropy available is more than 50% of the capacity, that means storage cache is replenished.

8. Reboot the host (hard reset is not required). This will boot the host with only user entropy source.

B. If user has strict compliance to use only user entropy source, please reinstall the host and provide the entropy via installer script.
 

4. VOB ID: esx.audit.entropy.external.source.disconnected

Host is configured with external entropy source and it expects entropy API to be called from external entropy source within inactive_source_timeout. The host has not received any entropy API call (GET or ADD) within  inactive_source_timeout from external entropy source.
Please check network connectivity between external entropy source and VC. Please make sure external entropy source calls entropy API within inactive_source_timeout.
Possible reasons and resolution:

1. There is disruption in network/infrastructure components pushing external entropy – Please ensure infrastructure is sized to support the entropy consumption appropriately.

2. Not enough entropy bits  being sent by the client – increase the amount or frequency of the ADD api calls.  Refer Configure ESXi Entropy for api details.

5. VOB ID: esx.problem.entropy.config.error

Host is configured with external entropy source. A recent configuration change related to entropy cache size has caused this issue.
Please check VOB message for further details about what configuration change caused this issue.
Possible reasons and resolution:

1.  Recent cache size change caused this error – Please revert the recent cache size configuration change made for entropy on the host. And restart entropy daemon.